sus.botdocs

AI finding

robots.txt has a group for a crawler name no crawler list knows

AI Warning Severity: Medium

Part of the audit robots.txt allows a bot the server turns away. Finding id: ai.robotsRuleForUnknownBot.

What it costs you

A crawler follows the group that carries its own name. Rules under a name no crawler uses apply to nobody, so the crawler you meant follows the rules for * instead.

What to do

Look up the crawler's exact robots.txt name on the vendor's page and correct the User-agent line.

What it checks

  • Rule names a bot that does not exist

How to fix it

Correct the User-agent line

  1. Open the documentation of the vendor you meant and copy the crawler's robots.txt name exactly.
  2. Replace '[user agent]' on line [line] with it.
  3. If no such crawler exists, delete the group.

How to check: Fetch /robots.txt and run this rule again.

How to verify by hand

The vendor's own page that lists its crawler names.

What this relies on

The rule was meant for a real crawler and the name is misspelt.