llms.txt invites what robots.txt forbids

Checks

• Page in llms.txt is blocked for AI bots llms_txt_url_disallowed_for_ai_bots
• Page in llms.txt is broken or redirects llms_txt_url_broken_or_redirected
• Page in llms.txt is marked not to index llms_txt_url_noindex
• Llms.txt is older than the site structure llms_txt_older_than_site_structure

Findings

• llms.txt lists a URL that AI crawlers cannot fetch or should not use
• The site gained a section after llms.txt was last changed

What this relies on

The owner wants AI systems to read the pages listed in llms.txt.

How to verify by hand

Server or CDN logs: do AI bots fetch those URLs.