Testing robots.txt Properly, Rule by Rule
Enter a site and a URL, pick a crawler, and this tells you whether that crawler is allowed to fetch that URL — along with the exact rule that decided it and the line it sits on. You can also paste a robots.txt you have not published yet, which is the safer way to test a change.
Getting this right matters more than it looks, because robots.txt does not work the way most people assume. Rules are not applied in file order, a crawler does not read every group that mentions it, and an empty Disallow means the opposite of what it appears to say.
At a glance
- The longest matching rule wins, not the first one in the file.
- On an equal-length tie, Allow beats Disallow.
- A crawler obeys exactly one group and ignores the rest, including the * group.
- robots.txt controls crawling, not indexing — a blocked page can still appear in search results.
How to Use This Tool
Enter the website
Just the domain is enough. robots.txt always lives at the site root, so any path you paste is discarded before the request is made.
Enter the URL you want to test
A full URL or just the path. Query strings are included in the match, because rules can and often do target them.
Choose the crawler
Googlebot, Bingbot, GPTBot, ClaudeBot and others. The answer genuinely differs between them, because each obeys a different group in the same file.
Read the matched rule
The verdict comes with the rule that produced it and its line number, plus any other rules that also matched but lost. That is the part that tells you what to change.
Test changes before publishing
Tick the paste option and try a draft file against the same URLs. A robots.txt mistake can deindex a site, and it is a bad thing to discover in production.
Where Google's Own Tester Went
If you are here because the robots.txt Tester disappeared from Search Console, that is not your imagination. Google retired that tool at the end of 2023 and replaced it with a much simpler robots.txt report, which shows the file Google last fetched and whether it parsed, but no longer lets you test a specific URL against a specific crawler.
The replacement report is still useful for one thing this tool cannot do: it tells you what Google actually has on file, including how recently it fetched it and whether the fetch failed. If Google is behaving as though a rule you deleted is still in place, that report is where you find out it has a stale copy.
For testing a URL against a rule, though, you now need something else — which is what this is.
The Matching Rules Almost Everyone Gets Wrong
This is the part worth reading even if you never use the tool, because the mental model most people carry is wrong in ways that produce real mistakes.
- The most specific rule wins, measured by the length of the path pattern. Not the first rule, not the last. "Disallow: /blog/" and "Allow: /blog/public-post" can appear in either order and the Allow still wins for that one URL, because it is longer.
- On an exact length tie, Allow beats Disallow. This is the tiebreaker Google documents, and it is why a permissive rule can be written deliberately to override a restrictive one.
- A crawler uses one group only. If your file has a Googlebot group and a * group, Googlebot reads its own and ignores * entirely — it does not merge them. Rules you assumed applied to everything may apply to nobody once a named group exists.
- "Disallow:" with nothing after it allows the entire site. It reads like a block and does the opposite. "Disallow: /" is the one that blocks everything.
- Paths are case-sensitive. "Disallow: /Admin" does not block "/admin".
- "*" matches any run of characters and "$" anchors to the end. "Disallow: /*.pdf$" blocks PDFs but not "/file.pdf?download=1", because the query string means the path no longer ends there.
Blocking Is Not the Same as Deindexing
This is the single most expensive misunderstanding in the whole subject. robots.txt controls crawling. It does not control indexing, and the two come apart in a way that catches people out.
If a page is blocked in robots.txt but other sites link to it, Google can still index the URL. It will not have read the page, so the result shows no description and often just the URL — but it is in the index, which is usually the exact outcome the block was meant to prevent.
Worse, blocking a page in robots.txt makes it impossible for Google to see a noindex tag on that page. If you want a page out of the index, Google has to be allowed to crawl it in order to read the instruction telling it to leave. Blocking and noindexing at the same time is self-defeating.
The rule of thumb: use robots.txt to stop crawling of things not worth the crawl budget — faceted navigation, internal search results, infinite parameter combinations. Use noindex to keep a page out of search results. Use authentication for anything that genuinely must not be seen, because robots.txt is a request that well-behaved crawlers honour and nothing more.
Blocking AI Crawlers
A newer use of robots.txt is controlling whether AI companies may use your content for training or retrieval. These crawlers use their own user-agent names, so they need their own groups — a rule under Googlebot has no effect on them.
The names change as new services appear, and a group for a crawler that does not exist simply does nothing, so there is no cost to listing them. GPTBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended are the ones most sites consider.
Two things worth being clear-eyed about. First, this is voluntary — it works because those companies choose to honour it, and a crawler that ignores robots.txt is not stopped by it. Second, blocking a retrieval crawler can also remove you from AI answers that would have cited and linked to you, which is a trade rather than a pure win.
Mistakes That Take a Site Out of Search
- Shipping a staging site's robots.txt to production. "Disallow: /" on a live site is the classic catastrophic deploy, and it is silent — nothing errors, traffic just stops.
- Blocking CSS and JavaScript. Google renders pages, and a page it cannot style or run is a page it judges on a broken version of itself.
- Assuming a rule under "*" applies to Googlebot when a Googlebot group also exists. It does not.
- Using robots.txt to hide something private. The file is public, and listing a path in it is an announcement that the path exists.
- Blocking a page you also want deindexed, so Google never reads the noindex.
- Forgetting the Sitemap line. It costs nothing and is one of the ways search engines discover your sitemap.
What a Good robots.txt Looks Like
For most sites, short. The common failure is not an insufficiently clever robots.txt; it is an over-engineered one that blocks something important by accident.
A typical file allows everything, blocks a handful of genuinely useless paths — internal search, cart and checkout, parameter-heavy filter URLs — and names the sitemap. That is often the whole thing.
If you are building one from scratch, our Robots.txt Generator produces a valid file with the right syntax, and you can test the result here before publishing it. The two are meant to be used together.
After You Change It
Google caches robots.txt for up to 24 hours, so a change is not necessarily live for crawlers the moment you upload it. If you have just unblocked something, give it a day before concluding it did not work.
Check the file returns HTTP 200. A 404 means no restrictions, which is usually fine. A 5xx is the dangerous case: Google treats a server error on robots.txt as a signal to stop crawling the site entirely until it can read the file again.
Then confirm the effect rather than the intent. Our SEO Analyzer fetches a page the way a crawler would, and the HTTP Header Checker shows whether a URL also carries an X-Robots-Tag header — a header-level noindex is invisible in the HTML and catches people out regularly.
For pages you have deliberately blocked, the XML Sitemap Validator is worth a run too. Listing a blocked URL in your sitemap sends search engines two contradictory instructions about the same page.
Frequently Asked Questions
Where did the robots.txt Tester in Google Search Console go?
Google retired it at the end of 2023 and replaced it with a robots.txt report that shows the file Google last fetched and whether it parsed. That report no longer lets you test a specific URL against a specific crawler, which is what this tool does.
If two rules match the same URL, which one wins?
The one with the longest path pattern, regardless of the order they appear in the file. If both are the same length, Allow beats Disallow. This tool shows every rule that matched and which one won, so you can see the decision rather than guess at it.
Does blocking a page in robots.txt remove it from Google?
No. robots.txt stops crawling, not indexing. A blocked page that other sites link to can still be indexed, usually showing as a bare URL with no description. To remove a page from search results, allow crawling and use a noindex tag — Google has to be able to read the page to see the instruction.
Is robots.txt case-sensitive?
The paths are. "Disallow: /Admin" does not block "/admin". Directive names such as User-agent and Disallow are not case-sensitive, and neither are user-agent names.
How do I block ChatGPT or other AI crawlers?
Give each one its own group by user-agent name — GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended — with the paths you want blocked. It works because those companies choose to honour robots.txt; it is a request, not a technical barrier. Blocking retrieval crawlers can also cost you citations in AI answers.
Does "Disallow:" with nothing after it block anything?
No — an empty Disallow allows the entire site. It is the opposite of how it reads. "Disallow: /" is the directive that blocks everything, and confusing the two is how sites disappear from search.
How long does Google take to notice a robots.txt change?
Up to about 24 hours, since Google caches the file. If you have just unblocked something, wait a day before concluding the change did not take effect.
What happens if robots.txt returns a server error?
This is the dangerous failure. A 404 means no restrictions and is harmless. A 5xx tells Google it cannot determine the rules, and it responds by pausing crawling of the site until it can fetch the file successfully.
Related Tools
Robots.txt Generator
Build a valid robots.txt, then test it here before you publish it.
XML Sitemap Validator
Check your sitemap does not list URLs your robots.txt blocks.
HTTP Header Checker
Spot an X-Robots-Tag noindex that is invisible in the page HTML.
Advanced On-Page SEO Analyzer
Fetch a page the way a crawler would and see what it can actually reach.