sus.botdocs

Security finding

A path that robots.txt hides answers to anyone

Security Error Severity: High

Part of the audit robots.txt hides paths that are open to everyone. Finding id: security.robotsDisallowedPathAnswers.

What it costs you

robots.txt is public. It points a stranger at this path, and nothing stops them opening it. Search engines obey the file; people do not have to.

What to do

Put a login in front of the path, or remove what is there. This check runs on verified domains only and requests only the paths your own robots.txt names.

What it checks

  • Blocked admin or backup path is open

How to fix it

Require a login for the path in nginx

  1. If the area has its own login, make it required for every address under [path] and skip the block above.
  2. Otherwise use the block above, with [path] in place of /admin/. Create the password file outside the web root with: htpasswd -c <file> <user>
  3. Then remove the Disallow line; a protected path does not need one.

How to check: curl -sI https://www.example.com[path] | head -n 1 (use your own host; expect 401, 403 or 404)

Require a login for the path in Apache

  1. If the area has its own login, make it required for every address under [path] and skip the block above.
  2. Otherwise use the block above, with [path] in place of /admin/. Create the password file outside the web root with: htpasswd -c <file> <user>
  3. Then remove the Disallow line; a protected path does not need one.

How to check: curl -sI https://www.example.com[path] | head -n 1 (use your own host; expect 401, 403 or 404)

Protect the path or empty it

  1. If the content is not needed on the web server, delete it or move it out of the web root.
  2. If it is needed, require a login for every address under [path], in the application or in the web server.
  3. Then remove the Disallow line; a protected path does not need one.

How to check: curl -sI https://www.example.com[path] | head -n 1 (use your own host; expect 401, 403 or 404)

How to verify by hand

Search Console showing the path indexed; logs showing unknown clients requesting it.

What this relies on

A path hidden from crawlers was meant to be private; robots.txt is not access control.