AI finding
A crawler is served different content from a browser
AI Warning Severity: High
Part of the audit Bots and people are served different content. Finding id: ai.botVsBrowserContent.
What it costs you
What the crawler reads is what gets indexed and quoted. If it reads a consent wall, a challenge page or another language, that is what search and AI results know of the page.
What to do
Serve crawlers the same page as visitors. Let them past the consent wall and do not redirect them by location.
What it checks
- Text differs between bots and visitors
- Bots get a consent wall or challenge page
- Bots get a different language or region
- Text shown only to bots
- Bots get only the paywall preview
How to fix it
Show the consent banner on top of the content
- Show the banner as a layer over the page. Do not replace the page content with it.
How to check: curl -A with the crawler's user agent and search the output for a sentence from the article. Then run this rule again.
Decide what crawlers get behind the paywall
- List the crawlers that should read the full text.
- Serve those the full text and mark the paywalled part in the page's structured data as the search engine's own documentation describes.
How to check: curl -A with the crawler's user agent and compare the output with the page a subscriber sees.
Check the bot settings of the firewall or CDN
- Find what answered with the challenge: a CDN, a web application firewall or a security plugin.
- Allow the crawlers you want there. We do not name the setting, because it differs by product.
How to check: Run this rule again. With server logs connected, the real crawler shows requests answered with 200.
Stop redirecting by location or language header
- Remove the redirect by IP address or Accept-Language.
- Link the language versions to each other and let the visitor choose.
How to check: curl -A with the crawler's user agent from another country and check the lang attribute of the page that comes back.
Compare the two versions and remove the difference
- Fetch the page with the crawler's user agent and with a browser, and compare the text.
- Text only crawlers get: remove it or show it to everyone.
- Text only browsers get: render it on the server, so it is in the response.
How to check: curl -A with the crawler's user agent and compare the output with the page in a browser. Then run this rule again.
A real case
A site's age check let only the Googlebot user agent through, so Google's inspection crawler got the age wall, which carried noindex, and Search Console reported every page as excluded. A test as Googlebot looked healthy, and the owner wrote that it cost months of Google indexing.
Thresholds
| Name | Default | Meaning |
|---|---|---|
min_bot_text_share | 0.8 | A crawler that gets less than this share of the browser's text (0 to 1) is reported. |
max_bot_text_share | 1.2 | A crawler that gets more than this share of the browser's text is reported: text served to crawlers only. |
How to verify by hand
A person; and the cached or quoted text in search and AI results.
What this relies on
Differences are accidental (consent walls, geo rules, bot protection), not deliberate cloaking.