SEO finding
Rules were added to or removed from robots.txt since the last fetch
SEO Warning Severity: Medium
Part of the audit robots.txt. Finding id: seo.robotsTxtRulesChanged.
What it costs you
One line in this file can close the whole site to search engines, or open an area you kept out on purpose. Such changes often arrive with a release and nobody notices for weeks.
What to do
Confirm the change was intended. If not, restore the previous line.
What it checks
- Rules changed since last check
How to fix it
Confirm or revert the change
- Ask who deployed the change to robots.txt and why.
- If it was not intended, put the previous version back. The diff shows which lines to restore.
How to check: Fetch /robots.txt and compare it with the previous version.
A real case
A development team released a robots.txt that blocked the production site, and the owner lost all organic traffic, 85 to 90% of the site's total. Almost two months later it had not come back.