SEO finding
Almost every page of the site asks search engines not to index it
SEO Error Severity: High
Part of the audit Indexability. Finding id: seo.noindexOnEveryPage.
What it costs you
The whole site stays out of search results, or drops out of them over the next weeks.
What to do
A noindex on nearly every page comes from one place: a setting of the CMS or site builder, a line in the server or CDN configuration that adds an X-Robots-Tag header, or a template. It is often left over from a test site. Remove it there if the site should be found.
What it checks
- Every page tells search engines to skip it
How to fix it
Remove the site-wide noindex
- Open [example] and look at the page source for <meta name="robots"> and at the response headers for X-Robots-Tag. The finding says how many pages have it in the header.
- Header: find the line that adds X-Robots-Tag in the web server, the CDN or the hosting platform settings and remove it, or limit it to the paths that should stay out of search.
- Meta tag on every page: it comes from the template or from a setting of the CMS or site builder that hides the site from search engines. Turn that setting off, or remove the tag from the shared template.
- If the site moved from a test address, check that the test configuration was not copied along.
How to check: curl -sI [example] shows no X-Robots-Tag with noindex and the page source has no noindex. Recrawl the site and run this rule again.
Thresholds
| Name | Default | Meaning |
|---|---|---|
min_pages | 5 | Only judge sites with at least this many crawled HTML pages that answer 200. Same as seo.wordpressHidesSiteFromSearch; a starting value. |
min_share | 0.9 | Report when at least this share of those pages carries noindex, in the meta tag or the X-Robots-Tag header. Same as seo.wordpressHidesSiteFromSearch; a starting value. janreges/siteone-crawler reports the site-wide case as one critical finding (competitors/github/janreges-siteone-crawler.md). |