What AI crawlers actually see: a ten-site study of server-side structured data

Put plainly, the machine-readable description of each brand, the JSON-LD that tells a system who the company is, what it sells and how it connects to the rest of the web, was absent from the first response the server sent. It appeared only once a full browser had downloaded the page, run its JavaScript and assembled the document in memory. A person with a browser would never notice. A system that reads the raw response, and never runs that JavaScript, sees a page with no structured data at all.

This was not one stray template or a single misconfigured tag. It held across all ten sites in the corpus, which spanned different sectors and different technology stacks. Where structured data existed at all, it appeared only once a full browser had downloaded the page, run its JavaScript, and assembled the document in memory. On several sites it did not exist in either view.

The reason this matters is a quiet property of how retrieval works. Most AI crawlers fetch a page the way a plain HTTP client does: they request the HTML and read what comes back. They do not open a browser, they do not wait for scripts to run, and they do not reconstruct the page the way a person's screen does. Rendering JavaScript across the open web is slow and costly, so a great deal of retrieval happens on the raw response alone.

The crawlers behind ChatGPT, Claude, and Perplexity retrieved the raw response and did not execute JavaScript (Vercel crawler analysis, 2025; confirmed across independent tests through mid 2026). Google's surfaces, Gemini, Google AI Overviews, and Google AI Mode, draw on Google's rendered index, though rendering there is queued rather than guaranteed. Server-side structured data is the only version every engine can read. When an engine builds an answer about a company, the material it has to work with is whatever was delivered in that version. If the structured data was never there, it was never part of the evidence, and the brand is described from prose and links alone, its careful machine-readable summary simply not there to read.

This creates a gap that standard tooling hides. Schema validators and rich-result testers open the page in a full browser, run the JavaScript and then inspect the outcome. In that view the structured data is present and correct, so the tool reports a pass. The page looks compliant, and everyone moves on.

But the validator and the AI crawler are looking at two different documents. One waited for the browser to finish; the other read the raw response and left. A site can pass every schema validator it is pointed at and still serve nothing to the systems that actually decide citations. The check measures the rendered page; the answer engines read the page before it renders. That is the heart of the problem: the tool that certifies the markup and the systems that consume it are not looking at the same thing.

When structured data is missing from the raw view, a brand does not simply lose a tidy summary. The entity itself, the single resolvable thing an engine should recognise as the company, starts to come apart. Four failure modes recur.

Entity collapse: A multi-division or multi-offering brand gets resolved down to its most-cited part. The engine knows the brand for one thing, so its other divisions, product lines, or categories never surface in answers about the categories they actually compete in. Without per-division structured data in the raw response, nothing tells the engine those divisions are distinct authorities.

Authority leakage: recognition pools in the global or head-office version of the brand instead of its market entities, so a country-specific question resolves to the wrong market's page, with the wrong currency, availability, or contact details.

Entity fragmentation is when one organisation is split into several partial identities. A product line reads as separate from its parent, a regional site reads as unrelated to the global brand, a sub-brand floats free of the company that owns it. Each fragment carries a little of the real entity and none carries the whole, so no single, confident description of the business can form. The engine hedges, and hedged descriptions are the ones least likely to be cited.

Variant collapse: a product range collapses into a single model, so individual variants stop surfacing. One SKU stands in for the whole line, and questions about the other models return nothing or return the wrong one.

The useful part of this finding is that anyone can reproduce it on their own domain, and you do not need our corpus to do it. The test is to look at your site the way a non-JavaScript crawler does, then compare that with what a browser shows. Our guide on how to see your site like an AI crawler walks through the exact steps.

If the structured data you expect is missing from the raw response, that is the version the answer engines are most likely to read. The fix is not more markup in the browser; it is making sure the markup a crawler needs is in the server's first response, before any JavaScript runs.

The method was deliberately simple, so that it is easy to repeat. For each site we compared a raw HTTP fetch with the rendered page, and looked for server-side JSON-LD in the raw view. Nothing more elaborate than that comparison was needed to produce the result.

This research comes from entity resolution work developed through Growth.pro engagements.

The limitations are worth stating plainly. This is a small sample, n=10, drawn from a single window in mid 2026, so it is a snapshot rather than a trend line, and a larger or repeated study could move the picture. The corpus also skews towards large, well-resourced brands, the sites most likely to have invested in structured data in the first place, which makes the uniformity of the result more striking but less generalisable to the long tail. Treat the finding as a strong reason to check your own raw view, not as a precise population estimate.