SEO finding
robots.txt is missing, cannot be read, or does not name the sitemap
SEO Warning Severity: Medium
Part of the audit robots.txt. Finding id: seo.robotsTxtUnusable.
What it costs you
Crawlers follow the file as they read it, not as you meant it.
What to do
[fix]
What it checks
- Robots.txt not found
- Robots.txt has errors
- No sitemap named in robots.txt
How to fix it
Make robots.txt answer
- Open [url] and note the status ([status]).
- For a server error, find what handles the address: a static file, the CMS or a plugin. Repair it or replace it with a static file.
- If you want no rules, publish a file that allows everything, or let the address answer 404. Do not leave it on a server error.
How to check: Fetch [url]: it answers 200 as plain text, or 404.
Correct the line
- Open [url] and go to line [line].
- Correct the spelling of the directive or delete the line.
- Keep one directive per line, in the form Name: value, with every rule below a User-agent line.
How to check: Run this rule again: the line is no longer reported.
Name the sitemap in robots.txt
- Use the full URL of your sitemap or sitemap index, with protocol and host.
How to check: Fetch [url]: the Sitemap line is there.
Repair robots.txt
- Open [url] and check the status and the content.
- [fix]
How to check: Fetch [url]: it answers 200 as plain text. Run this rule again.
A real case
A site's missing pages answered with HTTP 500 instead of 404, so the request for its robots.txt failed too and Google stopped crawling the site. The home page dropped out of Google and the client's traffic fell 90%.