sus.botdocs

Security finding

A page that looks private is public, indexable or cacheable

Security Warning Severity: Medium

Part of the audit Pages meant to be private are published as public. Finding id: security.privatePagesExposed.

What it costs you

Staging sites show unfinished work and often run with weaker passwords and test data copied from production. Admin logins in search results invite password guessing. A private page stored by a shared cache can be served to the next visitor.

What to do

Put test systems behind a login, keep login, admin, search and export addresses out of the index, and send private pages with Cache-Control: private, no-store.

What it checks

  • Login or admin page in sitemap or indexable
  • Staging or test site public and indexable
  • Account page served without login
  • Private page cacheable by shared caches
  • Internal search or export URLs indexable

How to fix it

Put the staging host behind a password in nginx

  1. Create the password file with the htpasswd utility of the Apache HTTP Server distribution: htpasswd -c /etc/nginx/.htpasswd <user>. Any path outside the web root will do.
  2. A login keeps both visitors and search engines out; a robots.txt rule only asks the second group.

How to check: curl -sI [url] | head -n 1 (expect 401)

Put the staging host behind a password in Apache

  1. Create the password file with: htpasswd -c /etc/apache2/.htpasswd <user>. Any path outside the web root will do.
  2. A login keeps both visitors and search engines out; a robots.txt rule only asks the second group.

How to check: curl -sI [url] | head -n 1 (expect 401)

Send noindex for these addresses in nginx

  1. Also take the address out of the sitemap.

How to check: curl -sI [url] | grep -i -E 'HTTP/|x-robots-tag|cache-control|www-authenticate'

Send noindex for these addresses in Apache

  1. Also take the address out of the sitemap.

How to check: curl -sI [url] | grep -i -E 'HTTP/|x-robots-tag|cache-control|www-authenticate'

Put the test host behind a login

  1. Require a login for the whole host, or allow only your office and VPN addresses.
  2. A login keeps both visitors and search engines out; a robots.txt rule only asks the second group.

How to check: curl -sI [url] | head -n 1 (expect 401 or 403)

Keep the address out of the index

  1. Send the response header X-Robots-Tag: noindex for these addresses, and remove them from the sitemap.
  2. Do not block them in robots.txt as well. A crawler that may not fetch the page never sees the noindex.

How to check: curl -sI [url] | grep -i x-robots-tag

Fix the access check

  1. Open [url] in a private window. If it shows account content, the page does not check for a login.
  2. Make the application redirect to the login, or answer 401 or 403, when no one is logged in.

How to check: curl -sI [url] | head -n 1 (expect 302 to the login, 401 or 403)

Stop shared caches storing private pages

  1. Have the application send Cache-Control: private, no-store on account, login, admin and checkout pages.
  2. If a CDN or proxy overrides the header, exclude these paths from caching there.

How to check: curl -sI [url] | grep -i -E 'HTTP/|x-robots-tag|cache-control|www-authenticate'

A real case

Singapore's data protection commission found that a training company left a course registration page with 53 people's names open without a login; search engines indexed it and nobody noticed for more than a year. The company was found in breach of the law and received a formal warning.

Source: Personal Data Protection Commission Singapore, decision on SSA Group International, 2020

How to verify by hand

Search Console showing these URLs indexed or with impressions.

What this relies on

Login, admin, staging and account pages are not meant for search results.