AI finding
AI answers are tracked for this site while its pages are closed to AI systems
AI Warning Severity: High
Part of the audit AI access is blocked while AI visibility is the goal. Finding id: ai.blockedButWanted.
What it costs you
You measure how often AI answers cite the site and at the same time tell AI systems not to read or quote it. Citations and the visits they bring fall away over the following weeks.
What to do
Choose per crawler and per page: open it if you want to be cited, or keep it closed and stop counting on citations from it.
What it checks
- AI search bots blocked despite tracking AI answers
- Quoting banned on pages meant to be cited
- AI visits dropped after a rule change
- Page cited by AI is now blocked
How to fix it
Take the cited page out of the blocked path
- Narrow the Disallow rule on robots.txt line [line] so that it no longer matches [url], or add an Allow rule for the page.
How to check: Run this rule again after the next robots.txt fetch. Compare AI referral sessions 28 days later.
Remove the snippet restriction from the cited page
- Remove nosnippet or max-snippet:0 from the robots meta tag of [url].
- Check the X-Robots-Tag response header too: curl -sI [url].
How to check: View the source and the response headers of the page, then run this rule again.
Open the crawler in robots.txt
- Delete or narrow robots.txt line [line].
- Keep a separate group with Disallow for training-only crawlers if that is the intent.
How to check: Run this rule again after the next robots.txt fetch. Compare AI referral sessions 28 days later.
A real case
Researchers at cloro compared 1,058 much-cited domains: those that blocked OAI-SearchBot had a median ChatGPT citation rate of 0 against 0.394 for those that allowed it, and PerplexityBot blockers 0 against 1.167 in Perplexity. The authors call it an association, not proven cause.
Thresholds
| Name | Default | Meaning |
|---|---|---|
training_bot_kind | training | Value of ai_user_agents.bot_kind for crawlers that only collect training data. Blocking those does not stop a page from being cited. |
min_drop_pct | 30 | Fall in AI referral sessions, in percent of the previous 28 days, from which a new blocking rule is reported as a failure. |
How to verify by hand
AI referral sessions in analytics and citation counts before and after the change.
What this relies on
A site that tracks its AI visibility wants to be cited. Used only until the owner has answered the question above.