robots.txt allows a bot the server turns away

Checks

• Allowed bot is refused or challenged allowed_bot_gets_403_or_challenge
• Blocked bot still gets the page blocked_bot_gets_200_and_ignores_rule
• Same company bots treated in opposite ways same_vendor_search_bot_allowed_training_bot_blocked_or_reverse
• Rule names a bot that does not exist rule_for_a_bot_name_that_does_not_exist

Findings

• robots.txt lets AI crawlers in, but Cloudflare turns them away
• robots.txt has a group for a crawler name no crawler list knows
• What robots.txt says about an AI crawler differs from what the server does

What this relies on

robots.txt states the owner's policy; the firewall or CDN default was never aligned with it.

How to verify by hand

CDN or firewall logs for that user agent.