Skip to main content
sus.botdocs
  • SEO
  • AI
  • QA
  • Privacy
  • Accessibility
  • Security
  1. Docs
  2. Security
  3. robots.txt hides paths that are open to everyone

Security audit

robots.txt hides paths that are open to everyone

4 checks · 2 findings · scope: domain

Audit id: security.conflict.robots_reveals.

Checks

  • Blocked admin or backup path is open disallowed_admin_or_backup_path_returns_200
  • Blocked path listed in sitemap disallowed_path_listed_in_sitemap
  • Blocked path linked from public pages disallowed_path_linked_from_public_pages
  • Blocked path indexed anyway disallowed_path_indexed_anyway

Findings

  • A path that robots.txt hides answers to anyone
  • A path that robots.txt hides is in the sitemap, linked or indexed

What this relies on

A path hidden from crawlers was meant to be private; robots.txt is not access control.

How to verify by hand

Search Console showing the path indexed; logs showing unknown clients requesting it.

  • All audits
  • About sus.bot
  • Open the app
  • Contact support