Robots.txt Generator
Create a valid robots.txt with allow and disallow rules, crawler-specific groups and your sitemap.
Runs in your browser. Your input is processed on your device and is not uploaded to our server.
How to use the Robots.txt Generator
- Choose the default policy.
- Add the paths to block or allow, and any crawler-specific rules.
- Add your sitemap, then copy or download robots.txt and upload it to your site root.
robots.txt asks crawlers not to fetch certain paths. It is public and is not a security feature, so never rely on it to hide private content. It does not remove pages from search results either: use noindex for that. Do not block CSS or JavaScript that pages need to render. Test the result with the Robots.txt Tester.
Frequently asked questions
Where does the file go?
At the root of the host: https://www.example.com/robots.txt. Each subdomain needs its own.
Can I block AI crawlers?
You can add groups for crawlers such as GPTBot, which well-behaved bots respect. The presets add these for you.