Skip to main content
sus.botdocs
  • SEO
  • AI
  • QA
  • Privacy
  • Accessibility
  • Security
  1. Docs
  2. SEO
  3. robots.txt blocks a page that carries noindex

SEO audit

robots.txt blocks a page that carries noindex

3 checks · 2 findings · scope: page, domain

Audit id: seo.conflict.robots_vs_noindex.

Checks

  • Blocked page also marked not to be indexed disallowed_url_has_noindex
  • Blocked page hides its canonical or language tags disallowed_url_has_canonical_or_hreflang_that_cannot_be_read
  • Page blocked after being marked not to index noindex_added_then_blocked

Findings

  • robots.txt blocks a page whose canonical, hreflang or noindex can then not be read
  • robots.txt blocks a page that carries noindex

What this relies on

The owner wants the page out of the index. A blocked page cannot be fetched, so the noindex is never seen.

How to verify by hand

Search Console shows the URL as 'Indexed, though blocked by robots.txt' or it still gets impressions.

  • All audits
  • About sus.bot
  • Open the app
  • Contact support