Security finding
A path that robots.txt hides is in the sitemap, linked or indexed
Security Warning Severity: Medium
Part of the audit robots.txt hides paths that are open to everyone. Finding id: security.robotsPrivatePathPublicised.
What it costs you
The path was meant to stay out of sight, and the site itself advertises it. Anyone following the sitemap or the link reaches it, and a search for the right words can show it.
What to do
If the content is private, protect it with a login and take it out of the sitemap and the links. If it is public after all, remove the Disallow line.
What it checks
- Blocked path listed in sitemap
- Blocked path linked from public pages
- Blocked path indexed anyway
How to fix it
Decide: private or public
- Private: require a login for [group key]. Remove its addresses from the sitemap and remove the public links to it.
- Already indexed: a login makes the address drop out of the index over time. For speed, use the removal tool in Search Console.
- Public after all: delete line [line] of robots.txt.
- Do not rely on robots.txt for privacy. It is a request to crawlers and a signpost for everyone else.
How to check: Fetch the sitemap and search it for [group key]: not found. Recrawl: no public link to it.
How to verify by hand
Search Console showing the path indexed; logs showing unknown clients requesting it.
What this relies on
A path hidden from crawlers was meant to be private; robots.txt is not access control.