Robots.txt Generator
Build a robots.txt file in a few clicks, then test whether a URL would be crawled by Googlebot or another bot.
Runs in your browser — nothing is uploadedrobots.txt
Test a URL
How to use Robots.txt Generator
- Choose whether crawlers are allowed by default.
- List paths to block or allow, add your sitemap URL and choose whether to block AI crawlers.
- Copy or download robots.txt and upload it to your site’s root (example.com/robots.txt).
- Use the tester to check specific URLs.
Questions
Does robots.txt hide pages from Google?
No. It asks crawlers not to fetch pages, but a blocked URL can still appear in results if other sites link to it. Use a noindex meta tag (on a crawlable page) to keep a page out of search results.
Which rule wins when Allow and Disallow overlap?
Google uses the most specific (longest) matching rule; if they are equally long, Allow wins. The tester on this page follows the same logic, including * and $ wildcards.
Will blocking AI crawlers affect my Google rankings?
Blocking Google-Extended only stops Gemini/Vertex AI training; it does not affect Googlebot or Search. Other AI crawlers such as GPTBot and CCBot are unrelated to search ranking.