AI finding
robots.txt lets AI crawlers in, but Cloudflare turns them away
AI Warning Severity: Medium
Part of the audit robots.txt allows a bot the server turns away. Finding id: ai.cloudflareBlocksAiCrawlers.
What it costs you
AI assistants cannot read the pages, so they cannot quote or cite them, whatever robots.txt says.
What to do
Look at the site's bot and AI-crawler settings in Cloudflare and decide which crawlers to let through, so robots.txt and the edge say the same thing.
What it checks
- Allowed bot is refused or challenged
How to fix it
Make Cloudflare and robots.txt agree
- In the Cloudflare dashboard, open the security settings for bots and AI crawlers of this site.
- Allow the crawlers robots.txt allows, or block them in robots.txt as well, so both say the same.
How to check: Fetch a page with the crawler's user agent: it gets 200, or robots.txt now disallows it.