SEO finding
robots.txt blocks a page that carries noindex
SEO Warning Severity: Medium
Part of the audit robots.txt blocks a page that carries noindex. Finding id: seo.robotsBlocksNoindex.
What it costs you
The page can stay in search results, shown without a description, for as long as other pages link to it.
What to do
Remove the block, wait until the page has dropped out of the index, then block it again if you still want to.
What it checks
- Blocked page also marked not to be indexed
How to fix it
Let the noindex be read first
- Delete or narrow robots.txt line [robots line] so the page can be fetched.
- Keep the noindex in place.
- When the page is gone from the index, restore the block if crawling it is still a problem.
How to check: Inspect the URL in Search Console: the page is reported as not indexed because of its noindex, and no longer as 'Indexed, though blocked by robots.txt'.
A real case
A WordPress site owner blocked the whole site in robots.txt and added noindex through Yoast, and the site still appeared in Google, with the line 'A description for this result is not available because of this site's robots.txt'.
How to verify by hand
Search Console: the URL is reported as 'Indexed, though blocked by robots.txt', or still gets impressions.
What this relies on
The owner wants the page out of the index. A crawler that obeys the block never fetches the page, so it never sees the noindex.