Skip to main content
sus.botdocs
  • SEO
  • AI
  • QA
  • Privacy
  • Accessibility
  • Security
  1. Docs
  2. SEO
  3. XML sitemaps
  4. The sitemap lists URLs that should not be in it, or misses pages that should

SEO finding

The sitemap lists URLs that should not be in it, or misses pages that should

SEO Warning Severity: Medium

Part of the audit XML sitemaps. Finding id: seo.sitemapEntries.

What it costs you

The sitemap is your list of pages worth indexing. An entry that cannot be indexed wastes a crawl and makes the rest of the list less believable.

What to do

List every indexable, canonical page that answers 200 exactly once, and nothing else. Set lastmod to the date the content last changed, or leave it out.

What it checks

  • Sitemap lists broken or redirected pages
  • Sitemap lists hidden pages
  • Sitemap lists non-preferred addresses
  • Pages missing from the sitemap
  • Same URL in several sitemaps
  • Last changed dates look wrong

How to fix it

Generate the sitemap from the pages that are meant to rank

  1. Let the sitemap generator leave out pages that are noindex, redirected, removed or canonicalised to another URL.
  2. List each URL in one sitemap only.

How to check: Recrawl the site and run this rule again.

Add the missing page type to the generator

  1. Look at which pages are reported. When they share a type (a new post type, a new language, a new section), the generator leaves that type out.
  2. Add the type in the settings of the sitemap generator.
  3. If single pages are missing, check whether they are excluded by hand.

How to check: Fetch the sitemap: the pages are listed. Run this rule again.

Fill lastmod from the date the content changed

  1. Find where the sitemap is generated and which date it writes.
  2. Use the date the page content was last edited, not the time the file was built and not a planned publication date.
  3. Write the date with its time zone, or as a plain date.
  4. If no such date is stored, leave lastmod out.

How to check: Fetch the sitemap on two different days: lastmod only moves for pages that were edited.

A real case

Bing reported in February 2023 that lastmod values in sitemaps were set correctly in 79% of cases and wrongly in 18%, most often as the date the sitemap was generated instead of the date the page changed.

Source: Bing Webmaster Blog, February 2023

Thresholds

Thresholds for The sitemap lists URLs that should not be in it, or misses pages that should
NameDefaultMeaning
lastmod_tolerance_days7A lastmod date may be this many days later than the last change we detected on the page before it is reported.
  • All audits
  • About sus.bot
  • Open the app
  • Contact support