Robots.txt Validator
Validate your robots.txt file for syntax errors and best practices
In 30 seconds
- robots.txt controls crawling, not indexing. It can’t remove a page from Google.
Disallow: /underUser-agent: *blocks your entire site.- Never block the CSS and JavaScript your pages need to render.
A small file that can hide your whole site
Paste the contents of a robots.txt file and the validator checks it line by line, right in your browser. Nothing is uploaded. It reports:
- Errors: rules that won’t work, like a
Disallowbefore anyUser-agent, aUser-agentwith no value, a missing colon, or a relative Sitemap URL. - Warnings: lines that are valid but probably not doing what you think, like duplicate user-agent groups, unknown directives and
Crawl-delay. - Information: the sitemaps it found, and a nudge to add a
User-agent: *group if there isn’t one. - Statistics: how many user-agent groups, rules and sitemaps the file has.
To check a live site, open https://yourdomain.com/robots.txt, copy everything, and paste it in.
What robots.txt does, and what it doesn’t
It’s a plain text file at the root of your domain that tells crawlers which URLs they may request. It’s the main way to keep bots out of places that waste their time: internal search results, cart and checkout pages, endless filter combinations.
Watch out
robots.txt does not remove pages from Google. A blocked URL can still be indexed, without a description, if other sites link to it. To keep a page out, let it be crawled and add a noindex tag. Google has to crawl the page to see it.
It’s also not security. The file is public and badly behaved bots ignore it. Listing a secret path in robots.txt is a good way to advertise it.
The syntax in two minutes
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlUser-agentstarts a group.*means every crawler without its own group.Disallowblocks URLs that start with the path. An emptyDisallow:blocks nothing.Allowcarves an exception out of a blocked path. Google uses the most specific (longest) matching rule.*in a path matches anything, and$matches the end, soDisallow: /*.pdf$blocks PDFs.Sitemaplines need a full URL and can go anywhere in the file.
A crawler follows only the one group that best matches its name. Add a Googlebot group and Googlebot ignores the * group completely, so repeat any shared rules inside it.
The usual mistakes
Blocking the whole site
Disallow: / left over from a staging site is one of the most common reasons a site vanishes from search right after launch.
Quick fix
Remove it, or change it to an empty Disallow:.
Blocking CSS and JavaScript
Google renders pages like a browser. Block the files it needs and it may see a broken page and misjudge your content.
Relying on Crawl-delay
Google ignores Crawl-delay. Bing and some other crawlers respect it, and a big value can slow down how quickly they find new pages. Only use it if a crawler is genuinely hammering your server.
Rules before a user-agent
Every Allow, Disallow and Crawl-delay belongs to a group. Rules above the first User-agent line are ignored.
Wrong name or place
It must be robots.txt, lowercase, at the root of each host. /blog/robots.txt does nothing, and shop.example.com needs its own file.
Quick answers
Do I need a robots.txt file?
No. Without one, crawlers assume everything is allowed. Most sites still want one, if only to point at the sitemap and keep bots out of search and cart pages.
How do I block AI crawlers?
Add a group per crawler, like User-agent: GPTBot with Disallow: /. Check each company’s docs for current names, and remember it’s a request, not a lock.
How fast do changes take effect?
Google caches robots.txt for up to a day. After a fix, you can ask it to refetch from the robots.txt report in Search Console.