AI finding
An AI crawler cannot fetch or read the site's pages
AI Warning Severity: High
Part of the audit AI crawler access. Finding id: ai.crawlerBlocked.
What it costs you
An AI assistant that cannot fetch or read a page cannot quote it or link to it. Visitors who ask the assistant are sent to other sites.
What to do
Decide per crawler whether you want it. For the ones you want: allow them in robots.txt, let them through the firewall, and send the main content in the HTML.
What it checks
- AI search bots blocked in robots.txt
- AI training bots blocked in robots.txt
- AI bots blocked by firewall or error
- AI bots blocked by page or server setting
- Content missing when scripts are off
How to fix it
Let the crawler through the firewall or CDN
- The site is served through a CDN ({stack.cdn}). Blocks of AI crawlers are often set there and not on your server.
- Look at its bot and AI crawler settings and allow [user agent]. We do not name the setting, because it differs per CDN and plan.
How to check: Run this rule again. With server logs connected, [user agent] shows requests answered with 200.
Allow the crawler in robots.txt
- In robots.txt, remove the Disallow rules in the group for [user agent], or the rule under User-agent: * that catches it.
- To keep a training crawler out and a search crawler in, give each its own group.
How to check: Fetch /robots.txt and run this rule again.
Remove the meta tag or header that names the crawler
- Remove robots meta tags or X-Robots-Tag headers that name [user agent], unless they are meant.
How to check: curl -sI [example] and view the page source. Then run this rule again.
Render the main content on the server
- Turn on server-side rendering or prerendering for the templates this finding lists.
- Where that is not possible, put at least the main text in the HTML and let the script add the rest.
How to check: curl -s [example] and search the output for a sentence from the middle of the page.
Let the crawler through the firewall or CDN
- If the status is 403 or 429, ask whoever runs the firewall, CDN or security plugin to allow [user agent]. We could not detect what blocks it.
How to check: Run this rule again. With server logs connected, [user agent] shows requests answered with 200.
A real case
Researchers at cloro compared 1,058 much-cited domains: those that blocked OAI-SearchBot had a median ChatGPT citation rate of 0 against 0.394 for those that allowed it, and PerplexityBot blockers 0 against 1.167 in Perplexity. The authors call it an association, not proven cause.
Thresholds
| Name | Default | Meaning |
|---|---|---|
min_text_share | 0.5 | Share of the browser's text (0 to 1) a bot must receive. Below it the content counts as missing without JavaScript. |