Security finding
robots.txt names paths that look private
Security Info Severity: Low
Part of the audit Information exposure. Finding id: security.robotsListsSensitivePaths.
What it costs you
robots.txt is public and is the first file a curious stranger reads. The line tells them where to look. It keeps search engines out; it stops no one else.
What to do
Request [path] yourself in a private browser window. If it answers, put a login in front of it. A path that needs a login does not need a line in robots.txt.
What it checks
- Robots file reveals sensitive paths
How to fix it
Protect the path, then drop the line
- Open [path] without being logged in. A 401, a 403, a 404 or a redirect to the login is fine.
- If the content shows, require a login for everything under that path.
- Once it is protected, the Disallow line adds nothing; remove it so the file stops pointing at the path.
How to check: curl -sI https://www.example.com[path] | head -n 1 (use your own host; expect 401, 403, 404 or a redirect to the login)