SEO finding
Bots hit errors, or search engines do not fetch pages you want found
SEO Warning Severity: Medium
Part of the audit Search bot activity from logs. Finding id: seo.botCrawlFromLogs.
What it costs you
Crawlers spend their visits on errors, or do not reach pages you want found.
What to do
Send crawlers to pages that work, and link to the pages they miss.
What it checks
- Search bots hitting error pages
- Important pages search bots never visit
How to fix it
Stop sending bots to errors
- Find where [url] is still linked or listed: internal links, the sitemap, old redirects.
- If the address should work, repair it. For status 500 and above, read the server's error log for the same URL and time.
- If it is gone for good, remove the links and sitemap entries.
- If one bot causes most of the errors by guessing addresses, that is noise. Errors on addresses you link to are the ones to fix.
How to check: Run this rule again on the logs of the next 30 days.
Get the page crawled
- Add links to [url] from related pages that crawlers visit often.
- Check that the sitemap lists the page and that its date is correct.
How to check: Run this rule again on the logs of the next 30 days.
Thresholds
| Name | Default | Meaning |
|---|---|---|
min_error_hits | 10 | A URL is reported when one bot got an error on it at least this often in 30 days. |
How to verify by hand
The crawl statistics in Search Console for the same period.
What this relies on
The delivered logs cover every server and CDN that answers for the site, for the full 30 days.