UtilTools

Robots.txt Generator

Crawl rules, including the AI crawlers.

Robots.txt Generator builds a valid robots file from rules you choose, with presets for the major search engines and for the AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — that many sites now want to decide about explicitly. It validates the syntax and explains each directive.

Loading Robots.txt Generator…

Runs entirely on your device. No file or text you enter here is uploaded. How to verify it.

How to use Robots.txt Generator

  1. Choose which crawlers to allow or block.
  2. Add disallow and allow paths, and your sitemap URL.
  3. Copy or download the robots.txt.

Questions about Robots.txt Generator

Does robots.txt actually stop a crawler?
It is a request, not a barrier. Well-behaved crawlers honour it; hostile ones ignore it entirely. Nothing in robots.txt secures anything — use authentication for that, and never list a secret path there, since the file itself is public.
Does Disallow remove a page from Google?
No, and this catches people out. Disallowing a page stops it being crawled, not indexed — it can still appear as a bare URL from external links. To remove a page from results, allow crawling and use a noindex meta tag.
Should I block AI crawlers?
That is your call and it is a trade-off. Blocking GPTBot and ClaudeBot keeps your content out of training and out of the answers those assistants give — including answers that would have cited you. The generator presents both directions without pushing one.
Where does the file have to go?
At the root of the host, as https://example.com/robots.txt, served as plain text. It applies only to that hostname and port, so a subdomain needs its own file, and a robots.txt placed in a subfolder is ignored completely.
Which crawlers can it write rules for?
Seven search crawlers — Googlebot, Bingbot, DuckDuckBot, Slurp, Baiduspider, YandexBot and Applebot — and twelve AI ones, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot and meta-externalagent. Training crawlers and user-initiated ones are listed separately, because blocking one and blocking the other are different decisions.
Is Robots.txt Generator really free, with no limits?
Yes. There is no account, no daily cap, no file-size tier and no watermark, because there is no server cost to recover. The site is funded by clearly disclosed partner recommendations on the Partners page, never by metering the tools.
Are my files or text uploaded when I use Robots.txt Generator?
No. Everything is processed by JavaScript running in your browser, on your device. You can verify it: open your browser's developer tools, switch to the Network tab, and use the tool. You will see no request carrying your data — because there is nowhere for it to go.