AI audit

AI crawler access

6 checks · 2 findings · scope: page, domain

Audit id: ai.crawler_access.

Checks

AI search bots blocked in robots.txt search_bots_blocked_in_robots
AI training bots blocked in robots.txt training_bots_blocked_in_robots
AI bots blocked by firewall or error blocked_by_firewall_or_status
AI bots blocked by page or server setting blocked_by_meta_or_header
Llms.txt file present llms_txt_present
Content missing when scripts are off content_missing_without_javascript

Findings