Page indexing report · Search Console

Indexed, though blocked by robots.txt

Your robots.txt stops Google from reading the page, but Google indexed the URL anyway because other pages link to it.1 It’s a warning, and often harmless. Here’s when it matters, and the right way to get a page out of Google, which isn’t robots.txt.

Updated · Based on Google’s documentation, statements from Google staff and published studies · 11 sources

Check the URL first

The checker tests the URL against your live robots.txt and looks for a noindex. It can’t tell whether Google still has the URL indexed. URL Inspection in Search Console can.

Free, no sign-up. Checks status, noindex, robots.txt, canonical, content and sitemap.

In 30 seconds

  • robots.txt controls crawling, not indexing. Google calls it “not a mechanism for keeping a web page out of Google”.2
  • The URL can appear in results, but without a description.2
  • To remove it, unblock the page and add noindex. A blocked page’s noindex is never seen.3
  • For cart, search and filter URLs, it’s usually safe to leave.6

What it means

Google’s explanation: it “always respects robots.txt”, but that “doesn’t necessarily prevent indexing if someone else links to your page”.1 Googlebot never fetched the page. It indexed the address from links pointing at it.

The block applies at the crawl stage, but this status is about what Google shows in results.

Serve

Indexed pages can appear in search results. Being indexed is not the same as ranking; an indexed page can still get no clicks.

What goes wrong: The page is indexed but doesn’t match what people search for, or other pages answer it better.

Report statuses at this stage

Because Google can’t read the page, the result has no description.2 Google explains the missing snippet the same way: a rule such as robots.txt stopped Googlebot from reading the content.4 John Mueller has said Google then has to rely mainly on links to judge whether the URL is worth showing, and that it will prefer pages it can crawl.5 He has also said the average user won’t see such URLs.6

Is it a problem?

Usually fine

  • Cart, checkout and add-to-cart URLs
  • Internal search results and filter or sort parameters
  • URLs that only show up in a site: search

Worth fixing

  • A page you want to rank, blocked by mistake
  • Private or sensitive URLs that must not be in Google
  • Pages with a noindex that robots.txt hides
  • Blocked URLs listed in your XML sitemap

John Mueller in 2026, on add-to-cart URLs: “Blocking them with robots.txt is fine.”7

Find the cause

Answer a few questions to pick the right fix. Each cause is explained below.

Question 1Do you want this URL to appear in Google?

Links to blocked URLs

The underlying cause. Internal links, other sites or your own sitemap point at a blocked URL, so Google knows it exists.1 Cart buttons, search forms and filter links create these URLs at scale.

A rule that’s broader than intended

robots.txt rules match from the start of the path, so a short rule can block far more than you meant. The tester below shows which rule matches.

noindex hidden by robots.txt

Adding noindex and a robots.txt block together, to be safe, backfires. Google says that if the page is blocked, the crawler “will never see the noindex rule”, and the page can still appear in results.3

Platform defaults

Shopify’s default robots.txt blocks cart, checkout, account, and filtered or sorted collection URLs.11 Those landing here is expected.

Find the rule and pick the fix

Paste your robots.txt and a URL from the report to see which rule blocks it.

Blocked

Matched Disallow: /*?sort= in the User-agent: * group. It’s the longest rule that matches.

Runs in your browser with Google’s matching rules: the most specific user-agent group applies, the longest matching path wins, and Allow wins a tie.

Then pick what you want to happen to the URL:

It should rank

  1. Find the rule that blocks the URL with the tester above.
  2. Remove it, narrow it, or add an Allow for the path.
  3. In Search Console’s robots.txt report, request a recrawl of the file.9 Google otherwise caches it for up to 24 hours.8
  4. Inspect the URL and click Request indexing.
# Before: blocks /products and /pricing too
User-agent: *
Disallow: /p

# After: blocks only /p/...
User-agent: *
Disallow: /p/

How to fix it

  1. Export the URLs and sort them: should rank, should be removed, or harmless.
  2. Remove any of them from your XML sitemap that you don’t want indexed.
  3. Should rank: remove or narrow the robots.txt rule, request a robots.txt recrawl, then request indexing.
  4. Should be removed: add noindex (or delete the page), then remove the robots.txt rule so Google can see it. Google says robots.txt “is not the correct mechanism” for this.1
  5. Harmless: leave them.

Myths

Some top-ranking guides say these. They’re wrong.

Often saidWhat’s actually true
Blocking a page in robots.txt removes it from Googlerobots.txt isn’t a way to keep a page out of Google; blocked URLs can still appear in results.2
Shopify doesn’t let you edit robots.txtYou can, by adding a robots.txt.liquid template to your theme. Shopify calls it an unsupported customisation.11

How long it takes

Indexed, though blocked vs Blocked by robots.txt

Indexed, though blocked by robots.txtBlocked by robots.txt
Can Google crawl it?NoNo
Is it indexed?Yes, from links alone1No
Report sectionIndexed, flagged as a warningWhy pages aren’t indexed
To get it indexedRemove the robots.txt ruleRemove the robots.txt rule
To keep it outUnblock and add noindexNothing, unless it starts getting indexed

Questions

Is Indexed, though blocked by robots.txt bad?

Usually not. It’s a warning, not an error. For cart, search and filter URLs that nobody searches for, Google’s John Mueller has said not to worry about it. It matters when the URL is a page you want to rank, or one you need out of Google.

How do I remove a page that is indexed but blocked by robots.txt?

Remove the robots.txt block and add a noindex tag, or delete the page so it returns 404. Google has to crawl the page to see either. For urgent cases, the Removals tool hides it for about six months while you do that.

Why did Google index a page that robots.txt blocks?

robots.txt controls crawling, not indexing. If other pages link to a blocked URL, Google can index the URL from those links alone, without reading the page.

Can I use noindex and robots.txt together?

Not on the same URL. If robots.txt blocks the page, Google never sees the noindex, so the URL can stay indexed. Use one or the other: noindex to keep a page out of the index, robots.txt to stop crawling.

How do I fix this on Shopify?

Shopify’s default robots.txt blocks cart, checkout, account and some filtered collection URLs, and those are usually fine to leave. To change the rules, add a robots.txt.liquid template to your theme. Shopify Support doesn’t help with edits, so change only what you need.

What is the difference from Blocked by robots.txt?

Both mean robots.txt stops Google from crawling the URL. Blocked by robots.txt means the URL isn’t indexed. Indexed, though blocked means Google indexed it anyway, from links, and shows it without a description.

All Page indexing statuses

Sources

  1. Google, “Page indexing report”, Search Console Help
  2. Google, “Introduction to robots.txt”, Search Central, updated December 2025
  3. Google, “Block Search indexing with noindex”, Search Central, updated December 2025
  4. Google, “No page information in search results”, Search Console Help
  5. John Mueller (Google), Webmaster Central hangout, July 2019, reported by Search Engine Journal
  6. John Mueller (Google), LinkedIn, September 2024, reported by Search Engine Journal
  7. John Mueller (Google), Reddit, June 2026, reported by Search Engine Journal
  8. Google, “How Google interprets the robots.txt specification”, Search Central, updated August 2026
  9. Google, “robots.txt report”, Search Console Help
  10. Google, “Removals and SafeSearch reports tool”, Search Console Help
  11. Shopify, “Editing robots.txt.liquid”, Shopify Help Center