SEO finding
robots.txt blocks a page whose canonical, hreflang or noindex can then not be read
SEO Warning Severity: Medium
Part of the audit robots.txt blocks a page that carries noindex. Finding id: seo.blockedPageSignalsUnread.
What it costs you
The instruction on the page has no effect. The blocked URL can stay in search results without a description, and the page it points to does not get the credit.
What to do
Remove the block until crawlers have read the page, then decide whether you still need it.
What it checks
- Blocked page hides its canonical or language tags
- Page blocked after being marked not to index
How to fix it
Unblock, let the page be read, then block again if still wanted
- Remove or narrow the robots.txt rule that matches [url].
- Keep the [signal] on the page as it is.
- Wait until Search Console no longer lists the URL as indexed.
- Put the block back only if crawling the URL is still a problem.
How to check: Search Console URL inspection no longer says 'Indexed, though blocked by robots.txt' for [url].
A real case
A WordPress site owner blocked the whole site in robots.txt and added noindex through Yoast, and the site still appeared in Google, with the line 'A description for this result is not available because of this site's robots.txt'.
How to verify by hand
Search Console shows the URL as 'Indexed, though blocked by robots.txt' or it still gets impressions.
What this relies on
The owner wants the page out of the index. A blocked page cannot be fetched, so the noindex is never seen.