SEO finding
robots.txt blocks pages you want found, or files the pages need to render
SEO Error Severity: High
Part of the audit robots.txt. Finding id: seo.robotsTxtBlocksNeededUrls.
What it costs you
Crawlers that obey robots.txt cannot fetch these URLs.
What to do
Change the rule so it no longer matches URLs the site needs.
What it checks
- Blocks important parts of the site
- Blocks files needed to show pages
How to fix it
Narrow or remove the rule
- robots.txt line [line]: Disallow: [group key]
- Delete the line, or replace it with the narrower path you meant to block.
- To keep the rule but free one folder, add an Allow line with the longer path below the same User-agent line.
How to check: Fetch /robots.txt and confirm the change. Recrawl: the URLs are no longer reported as blocked.
Allow the files the pages need
- Replace the path with the folder that holds [url].
- The Allow path must be longer than the Disallow path ([group key]). The longer rule wins.
- Or delete line [line], if nothing under [group key] needs blocking.
How to check: Fetch /robots.txt and confirm the change. Recrawl: the files are no longer reported as blocked.
A real case
A development team released a robots.txt that blocked the production site, and the owner lost all organic traffic, 85 to 90% of the site's total. Almost two months later it had not come back.
Thresholds
| Name | Default | Meaning |
|---|---|---|
min_inlinks | 10 | A blocked page counts as important when it is in the sitemap or has at least this many internal links pointing at it. |
How to verify by hand
Search Console clicks and the URL inspection for one of the blocked URLs.
What this relies on
A page that is in the sitemap or linked from many pages is meant to be found. The robots.txt rule is the mistake.