FREE TOOL
robots.txt generator
Choose which search engines and AI crawlers may visit your site, and download a ready-made robots.txt.
WHAT IT DOES
About this tool
Build a robots.txt file: allow or block search engines, disallow folders, add your sitemap and block AI crawlers such as GPTBot, ClaudeBot and CCBot.
Everything runs right in your browser: your text and files are never uploaded or stored.
QUESTIONS
Frequently asked questions
Does robots.txt hide pages from Google?
No. It asks well-behaved crawlers not to visit pages, but a blocked page can still show up in search if other sites link to it. To remove a page from search results, allow crawling and add a noindex meta tag instead.
Should I block AI crawlers?
That's a business call. Blocking training bots like GPTBot or ClaudeBot keeps your content out of future training data. Blocking search bots like OAI-SearchBot or PerplexityBot also means your pages won't be cited in those AI answers, which can cost you traffic.
Where does the file go?
In the root of your domain, so it opens at https://your-domain.com/robots.txt. Each subdomain needs its own file. Test it in Google Search Console under Settings → robots.txt.
06 / LET'S GET ACQUAINTED
TELL US
WHAT YOUR
BUSINESS NEEDS
Describe the task. Together we'll
work out the next step.