AI finding
What robots.txt says about an AI crawler differs from what the server does
AI Error Severity: High
Part of the audit robots.txt allows a bot the server turns away. Finding id: ai.robotsVsBotResponse.
What it costs you
A crawler you allow in robots.txt but refuse at the firewall never reads the site, so the pages cannot be quoted by that vendor's products. A crawler you forbid but still serve takes content you meant to keep back.
What to do
Decide per crawler whether it may read the site, then make robots.txt and the firewall or CDN bot settings say the same.
What it checks
- Allowed bot is refused or challenged
- Blocked bot still gets the page
- Same company bots treated in opposite ways
How to fix it
Align the firewall or CDN with robots.txt
- The site is served through a CDN. Refusals and challenges for crawlers are usually decided there, not on your own server.
- Find the bot or firewall rule that matches [user agent] in the CDN's security settings. We do not name the setting because it differs by product and plan.
- Allow the crawlers robots.txt allows.
How to check: Run this rule again. With logs connected, look for status 200 for the allowed crawler.
Confirm the split per vendor
- Open the vendor's documentation of its crawlers and read what [user agent] and [other user agent] are for.
- Keep the rules if the split is meant. Otherwise give both crawlers the same rules in robots.txt.
How to check: Fetch /robots.txt and run this rule again.
Block the crawler where requests are answered
- Add a rule for [user agent] at the firewall, CDN or web server. robots.txt alone refuses nobody.
- Where the vendor publishes the addresses its crawler uses, compare them with the addresses in the logs to tell the real crawler from an impostor.
How to check: With logs connected, requests under that name are answered with 403.
Align the firewall or CDN with robots.txt
- Find what refuses [user agent]: a CDN, a web application firewall, a security plugin or a server rule.
- Allow the crawlers robots.txt allows.
How to check: Run this rule again. With logs connected, look for status 200 for the allowed crawler.
A real case
An IT provider switched on Cloudflare's AI crawler blocking for a client's site, and the setting also shut out Googlebot without any change to robots.txt. The site's pages fell out of Google's index and its shopping listings dropped, and recovery took about two weeks after the block was lifted.
How to verify by hand
CDN or firewall logs for that user agent.
What this relies on
robots.txt states the owner's policy; the firewall or CDN default was never aligned with it.