How robots.txt Works
Choose whether to allow all crawlers or restrict specific paths, list the directories you want blocked, add your sitemap URL, and this tool builds a valid robots.txt file you can copy straight to your server. The output updates as you type.
robots.txt is a plain text file at the root of your domain that tells crawlers which parts of the site they may request. It is the first file most crawlers look for, and one of the easiest files on a website to get catastrophically wrong.
What robots.txt does and does not do
robots.txt controls crawling — whether a bot requests a URL at all. It does not control indexing. This distinction causes more SEO damage than almost any other technical misunderstanding.
If you block a page in robots.txt, Google will not fetch it. But if other pages link to that URL, Google can still index it based on those links alone, showing it in results with no description because it was never allowed to read the page. Blocking a URL is therefore not a way to keep it out of search results.
To keep a page out of the index, use a noindex meta tag and allow crawling, so the crawler can actually see the directive. Blocking the page in robots.txt prevents Google from ever reading the noindex tag, which is exactly backwards from what people usually intend.
The directives you will use
- User-agent — which crawler the following rules apply to. An asterisk means all crawlers.
- Disallow — a path prefix crawlers should not request. Disallow: /admin/ blocks everything under /admin/.
- Allow — an exception carving a path out of a broader Disallow rule.
- Sitemap — the full URL of your XML sitemap. Independent of user-agent blocks and can appear anywhere in the file.
- An empty Disallow: means nothing is blocked, which is equivalent to allowing everything.
The mistake that takes a site out of Google
A single line — Disallow: / — blocks your entire site from every crawler. It is a legitimate setting for a staging environment, and it is also the single most common way a live site accidentally disappears from search.
The usual sequence is that a site is built on staging with a blanket block, then pushed to production with the file copied across unchanged. Rankings do not vanish immediately, which makes the cause harder to spot weeks later when traffic has collapsed.
Always check your live robots.txt after any launch, migration, or hosting change. It takes ten seconds: open yourdomain.com/robots.txt in a browser and read it.
What is actually worth blocking
Less than most people assume. Modern search engines are good at ignoring low-value pages on their own, and an over-aggressive robots.txt causes more problems than it solves.
Sensible candidates are admin areas, internal search results pages, cart and checkout flows, and faceted-navigation URL parameters that generate near-infinite combinations. On a large ecommerce site, parameter URLs are the genuine crawl-budget problem worth addressing.
Do not block CSS or JavaScript files. Google renders pages to evaluate them, and blocking the resources needed to render leaves it seeing a broken layout. This was common advice years ago and is now actively harmful.
- Block: /admin/, /wp-admin/, internal search results, cart and checkout, duplicate parameter URLs.
- Do not block: CSS, JavaScript, images used in content, or any page you want indexed.
- Never block: your entire site on production, unless you genuinely want it out of search.
Adding your sitemap
Including a Sitemap line is the simplest way to point crawlers at your XML sitemap, and it works for every crawler that reads robots.txt rather than only those you submit to manually. Use the full absolute URL, including the protocol.
You can list several sitemap lines if you split sitemaps by content type. If you do not have one yet, the Sitemap Generator will build valid XML from a list of URLs.
Where the file goes and how to check it
robots.txt must sit at the root of the domain — https://example.com/robots.txt. A file in a subdirectory is simply ignored. Each subdomain needs its own file, so blog.example.com does not inherit example.com's rules.
Rules are also protocol and host specific, which is a common gotcha during an HTTPS migration: the file served over HTTPS is the one that applies to HTTPS URLs. After uploading, open the URL directly to confirm it serves as plain text, then use Search Console's robots.txt report to confirm Google reads it as you intended.
It is worth re-checking after the fact too. If pages you expect to rank are missing, run one through the On-Page SEO Analyzer — a page blocked from crawling or carrying a noindex tag will show up there quickly.
Frequently asked questions
Does robots.txt stop a page appearing in Google?
No. It stops the page being crawled, but Google can still index a blocked URL based on links pointing to it, showing it without a description. To keep a page out of results, allow crawling and use a noindex meta tag so the crawler can read the directive.
Where does the robots.txt file go?
At the root of the domain, served at https://yourdomain.com/robots.txt. A file placed in a subdirectory is ignored entirely, and each subdomain needs its own file.
Do I need a robots.txt file at all?
Not strictly. If you have nothing to block, crawlers will simply crawl everything they can find. A minimal file that allows everything and points to your sitemap is still worth having, since it avoids 404s on the request and gives you somewhere to add rules later.
Should I block CSS and JavaScript?
No. Google renders pages to assess them, and blocking the files needed to render means it sees a broken page. This was once common advice but is now actively harmful.
Do all crawlers respect robots.txt?
The major search engines do. It is a voluntary standard, so malicious scrapers routinely ignore it. Never rely on robots.txt to protect sensitive content — use authentication for that.