SEO finding
A page is kept out of search, or its indexing signals contradict each other
SEO Warning Severity: High
Part of the audit Indexability. Finding id: seo.indexabilitySignals.
What it costs you
A page that is kept out of the index gets no search traffic. Contradicting signals leave the decision to the search engine.
What to do
For each page decide: should it be found in search or not. Then make robots.txt, the header, the meta tag, the canonical and the sitemap all say the same.
What it checks
- Page tells search engines to skip it
- Server tells search engines to skip page
- Page tells search engines not to follow links
- Page blocked from search engines
- Skipped page listed in sitemap
- Page points elsewhere and is also skipped
How to fix it
Make the signals agree
- Page should be found: remove noindex from the meta tag and from the X-Robots-Tag header, and remove the robots.txt rule that blocks it.
- Page should stay out: keep the noindex, take the URL out of the sitemap and remove the canonical that points elsewhere.
- Page is a copy of another: keep the canonical and remove the noindex.
- A noindex left over from a test site is usually one setting or one header line for the whole site. Look there first when many pages are listed.
How to check: curl -sI [url] shows no X-Robots-Tag you did not expect; the page source shows the robots meta tag you chose. Then run this rule again.
A real case
A web agency found that a client's WordPress site had been left on 'Discourage search engines', which keeps every page out of search, for a few years. Search traffic recovered once the box was unchecked.