AI finding
llms.txt lists a URL that AI crawlers cannot fetch or should not use
AI Error Severity: Medium
Part of the audit llms.txt invites what robots.txt forbids. Finding id: ai.llmsTxtEntryUnreachable.
What it costs you
llms.txt is your list of pages for AI systems. An entry they cannot fetch or are told not to use is a wasted entry, and it makes the rest of the file less credible.
What to do
For each entry, either make the page reachable for AI crawlers or take it out of llms.txt.
What it checks
- Page in llms.txt is blocked for AI bots
- Page in llms.txt is broken or redirects
- Page in llms.txt is marked not to index
How to fix it
Replace or delete the entry
- Status [status]: replace the entry on line [line] with the URL that answers 200, or delete it.
How to check: Fetch /llms.txt and request each listed URL, then run this rule again.
Remove the noindex or delete the entry
- If the page should be used: remove noindex from its robots meta tag or X-Robots-Tag header.
- If not: delete line [line] from llms.txt.
How to check: Fetch the page and /llms.txt, then run this rule again.
Narrow the robots.txt rule or delete the entry
- If AI crawlers should read the page: narrow or delete robots.txt line [robots line] ([blocking rule], user agent [bots affected]).
- If not: delete line [line] from llms.txt.
How to check: Fetch /llms.txt and /robots.txt, then run this rule again.
A real case
A user found 18 broken links among the 1,733 entries of vercel.com/llms.txt: eight were a generated address ending in 'undefined/no-summary' and the rest were pages that had moved while llms.txt kept the old paths. Any AI agent following those entries got a 404.
How to verify by hand
Server or CDN logs: do AI bots fetch those URLs.
What this relies on
The owner wants AI systems to read the pages listed in llms.txt.