AI Crawlers, and Why Blocking Them Is Rarely One Decision
Enter a domain and this reads its robots.txt, then works out what each of sixteen AI agents is actually allowed to do — following the same matching rules Google uses, so a rule aimed at one crawler is not misread as applying to another. It also checks for an llms.txt.
The reason this needs its own tool rather than a line in a robots.txt tester is that these agents are not interchangeable. Some index your pages so an assistant can cite and link you. Others absorb your content into a model with no attribution. Blocking the first group while allowing the second is a common mistake, and nothing in the file makes it obvious.
At a glance
- Search agents cite and link you. Training agents do not. They need different decisions.
- Google-Extended has no effect on Google Search — it only controls Gemini training.
- Blocking GPTBot does not stop ChatGPT reading a page a user asks about. That is ChatGPT-User.
- robots.txt is a request, not enforcement. Only server-side blocking is actually enforced.
How to Use This Tool
Enter a domain
The site's robots.txt is fetched and every agent is tested against it. Enter a specific URL instead to test that path rather than the homepage.
Read the AI search group first
These are the agents that send traffic back. If they are blocked and the training agents are not, the rules are almost certainly backwards.
Check what is explicit
Agents marked as named in robots.txt were decided on. Everything else inherited a wildcard rule, which means the access was never actually chosen.
Expand any row for the consequence
Each agent explains what blocking it does and, more usefully, what it does not do.
Three Kinds of Agent, Three Different Decisions
Lumping these together is what produces rules people later regret. The categories behave differently enough that a single policy across all of them is almost always wrong in one direction or the other.
- Search agents — OAI-SearchBot, Claude-SearchBot, PerplexityBot. They index your pages so an assistant can surface, cite and link them. This is the closest thing to traditional search traffic in an AI interface, and blocking these removes you from it entirely.
- Training agents — GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent. They collect content for model training. No citation, no link, no traffic. This is the group most people actually mean when they say they want to block AI.
- On-demand fetchers — ChatGPT-User, Claude-User, Perplexity-User. They read one page because a user asked about it. Not bulk crawling and not training. Blocking these means a person who explicitly wanted to read your page through an assistant is told it could not be read.
The Mistakes That Come From Copied Rules
Most AI blocking rules in the wild were pasted from a blog post, and the two most common errors follow directly from that.
The first is blocking Google-Extended in the belief it affects Google Search. It does not, at all. Google-Extended controls whether your content trains Gemini and grounds its answers, and has no bearing on crawling, indexing or ranking in Google Search. It is confused with Googlebot constantly because it sits in the same file, and the confusion runs both ways — people block it fearing SEO damage that cannot happen, and people avoid blocking it for the same imaginary reason.
The second is blocking GPTBot and assuming ChatGPT can no longer see the site. GPTBot is the training crawler. When a user pastes your URL into ChatGPT and asks about it, that fetch comes from ChatGPT-User, a separate agent with a separate rule. Blocking one does nothing to the other, and blocking GPTBot alone is a coherent position rather than an incomplete one: no training, but users can still read you.
The third, and the one that costs the most, is blocking every agent with an AI-sounding name. That includes the search agents, which are the ones that would have cited and linked you. Our Robots.txt Tester will show you exactly which rule is matching if a result here is not what you expected.
What Blocking Actually Achieves
Two limits worth being clear about before deciding anything.
robots.txt is a request. Well-behaved crawlers honour it and nothing compels the rest. Operators who ignore it are not stopped by a line in a text file, and the ones widely reported as ignoring it are the ones you would most want to stop. Actual enforcement means blocking by user agent or IP at the server or CDN, which is a different piece of work.
And blocking is prospective only. It affects future crawls. Content already collected is already in the training sets of models that have shipped, and no directive retrieves it.
So the honest framing: blocking limits future collection by cooperative crawlers. That is a real thing and worth doing if you want it. It is not a removal mechanism, and it is not enforcement.
Who Should Actually Block
It depends on how your site makes its money, and the answer genuinely differs.
Publishers whose revenue depends on people arriving at the page have the strongest case. An assistant that answers the question using your article, without a click, has taken the value and left the cost. Several large news organisations block nearly every AI agent for exactly this reason.
Sites selling something have close to the opposite position. Being cited by an assistant is a recommendation to someone already asking about your category. Blocking the search agents removes you from a surface where competitors will still appear.
Documentation and support content usually wants to be read. Being the source an assistant quotes when someone asks how to use your product is straightforwardly good, and blocking it just means the assistant answers from somewhere less accurate.
The reasonable middle for most sites: allow the search agents, decide on the training agents according to how you feel about it, and allow the on-demand fetchers, since those are real people who asked for your page specifically.
About llms.txt
A proposed convention: a markdown file at /llms.txt pointing AI systems at your most useful pages, on the theory that a curated list is more useful than a crawl.
It is worth being straight about its status. It is a proposal, not a standard. No major AI operator has committed to reading it, and there is no evidence yet that having one changes how any assistant treats a site. This checker reports whether you have one because it is cheap to know, not because its absence is a problem.
If you want one, it costs an afternoon and might matter later. If you would rather wait for an operator to actually commit to it, that is an equally defensible read of the same situation. What it is not is an urgent gap, whatever the volume of posts saying otherwise.
Where This Fits
AI access is one part of how your site is read by machines. Our Robots.txt Tester covers the same file for traditional search crawlers and shows exactly which rule matches a given path, and the Indexability Checker combines robots.txt with the meta and header directives that decide ordinary indexing.
There is also a content angle worth noting. Assistants quote pages that state things clearly, and structured data helps them work out what a page is describing. Our Schema Markup Validator checks that markup — it is read by AI crawlers as well as search engines, which is one of the few mechanisms for asserting facts about your content rather than hoping they are inferred.
Frequently Asked Questions
Does blocking Google-Extended affect my Google rankings?
No. Google-Extended controls only whether your content trains Gemini and grounds its answers. It has no effect on Googlebot, on indexing, or on ranking in Google Search. The two are confused constantly because they appear in the same file.
If I block GPTBot, can ChatGPT still read my site?
Yes, when a user asks about a specific page. That fetch comes from ChatGPT-User, a different agent with a different rule. GPTBot is the training crawler. Blocking GPTBot alone is a coherent position: no training, but users can still read you.
Which AI crawlers should I allow?
For most sites, allow the search agents — OAI-SearchBot, Claude-SearchBot, PerplexityBot — because those cite and link back. The training agents are a judgement call. The on-demand fetchers are worth allowing, since those requests come from people who explicitly wanted your page.
Does robots.txt actually stop AI crawlers?
It stops the ones that choose to honour it. Nothing compels the rest, and the operators most often accused of ignoring it are the ones you would most want to stop. Real enforcement means blocking by user agent or IP at the server or CDN.
Can I remove my content from models already trained?
No. Blocking is prospective — it affects future crawls only. Anything already collected is in the training sets of models that have shipped, and no robots.txt directive retrieves it.
Do I need an llms.txt?
No. It is a proposed convention rather than a standard, no major operator has committed to reading it, and there is no evidence yet that it changes how assistants treat a site. Cheap to add if you want to; not a gap if you do not.
Will blocking AI crawlers hurt my SEO?
Not for traditional search. None of these agents is Googlebot or Bingbot, and blocking them has no effect on ordinary indexing. What it costs is visibility inside AI assistants, which is a separate surface — and only if you block the search agents rather than just the training ones.
What does 'named in robots.txt' mean here?
That a group in your file lists that agent by name, so its access was an actual decision. Agents without that label inherited a wildcard rule, which usually means nobody chose their access either way.
Related Tools
Robots.txt Tester
Test any path against any crawler and see exactly which rule matches.
Indexability Checker
Combine robots.txt with the meta and header directives for ordinary indexing.
Schema Markup Validator
Structured data is read by AI crawlers as well as search engines.
Robots.txt Generator
Write the rules, then check the result here.