Robots.txt Validator

Validate your robots.txt file for syntax errors and best practices

In 30 seconds

  • robots.txt controls crawling, not indexing. It can’t remove a page from Google.
  • Disallow: / under User-agent: * blocks your entire site.
  • Never block the CSS and JavaScript your pages need to render.

A small file that can hide your whole site

Paste the contents of a robots.txt file and the validator checks it line by line, right in your browser. Nothing is uploaded. It reports:

To check a live site, open https://yourdomain.com/robots.txt, copy everything, and paste it in.

What robots.txt does, and what it doesn’t

It’s a plain text file at the root of your domain that tells crawlers which URLs they may request. It’s the main way to keep bots out of places that waste their time: internal search results, cart and checkout pages, endless filter combinations.

Watch out

robots.txt does not remove pages from Google. A blocked URL can still be indexed, without a description, if other sites link to it. To keep a page out, let it be crawled and add a noindex tag. Google has to crawl the page to see it.

It’s also not security. The file is public and badly behaved bots ignore it. Listing a secret path in robots.txt is a good way to advertise it.

The syntax in two minutes

User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

A crawler follows only the one group that best matches its name. Add a Googlebot group and Googlebot ignores the * group completely, so repeat any shared rules inside it.

The usual mistakes

Blocking the whole site

Disallow: / left over from a staging site is one of the most common reasons a site vanishes from search right after launch.

Quick fix

Remove it, or change it to an empty Disallow:.

Blocking CSS and JavaScript

Google renders pages like a browser. Block the files it needs and it may see a broken page and misjudge your content.

Relying on Crawl-delay

Google ignores Crawl-delay. Bing and some other crawlers respect it, and a big value can slow down how quickly they find new pages. Only use it if a crawler is genuinely hammering your server.

Rules before a user-agent

Every Allow, Disallow and Crawl-delay belongs to a group. Rules above the first User-agent line are ignored.

Wrong name or place

It must be robots.txt, lowercase, at the root of each host. /blog/robots.txt does nothing, and shop.example.com needs its own file.

Quick answers

Do I need a robots.txt file?

No. Without one, crawlers assume everything is allowed. Most sites still want one, if only to point at the sitemap and keep bots out of search and cart pages.

How do I block AI crawlers?

Add a group per crawler, like User-agent: GPTBot with Disallow: /. Check each company’s docs for current names, and remember it’s a request, not a lock.

How fast do changes take effect?

Google caches robots.txt for up to a day. After a fix, you can ask it to refetch from the robots.txt report in Search Console.