This page documents how I ran the test behind Schema markup gives AI assistants nothing plain text doesn’t. It covers the test pages, the models and versions, the exact prompts, trial counts, scoring and statistics.
The test pages
Every page describes a fictional company and carries one fact a user might ask about. There are five fact types: price, numeric product spec, address, author and publication date. Because the companies are invented, a correct answer can only come from the test page.
Each fact appears in five versions of the page, and each version comes in two schema types (Article and FAQPage):
| Version | Where the fact lives |
|---|---|
| A | Body text only (baseline) |
| B | Body text plus the fact in JSON-LD |
| C | Body text plus the same fact restated in prose, matched in length to B’s JSON-LD |
| D1 | Only in JSON-LD, either valid markup or deliberately broken markup (invented @context and @type) |
| D2 | Body says one value, JSON-LD says a different, false value |
D1 is split into valid and broken markup, which gives six variants per fact. With five fact types and two schema types, that’s 60 pages.
Layer 1: the model (21,600 trials)
I built the retrieval step myself so I could control what each model saw. This layer uses a larger set of fictional companies than the live site: 20 questions, 4 per fact type, each with its own target page in all 12 versions. The 60 live pages in layers 2 and 3 use a separate set of five companies, one per fact type, built the same way.
- Corpus: 20 questions, each with 1 target page and 9 distractor pages (10 documents per question).
- Chunking: 512 tokens with 64 tokens of overlap. The embedded text includes the JSON-LD, so schema can affect retrieval the same way prose can.
- Embedder:
nomic-embed-text, held constant for every model, so retrieval is identical across models. - Retrieval: top 5 chunks passed to the model.
- Generation: temperature 0, seed 42.
Models:
| Model | Version | How it ran | Trials per cell |
|---|---|---|---|
| Gemma 3 12B | gemma3:12b | Local (Ollama) | 30 |
| Phi-4 14B | phi4:14b | Local (Ollama) | 30 |
| Claude Sonnet 5 | claude-sonnet-5 | Anthropic API | 10 |
| GPT-5.5 | gpt-5.5 | OpenAI API | 10 |
| Gemini Flash | gemini-flash-latest | Gemini API | 5 |
| Gemini Pro | gemini-pro-latest (gemini-3.1-pro-preview) | Gemini API | 5 |
That’s 20 questions × 2 schema types × 6 variants × trials per cell: 14,400 local trials and 7,200 hosted trials, 21,600 in total. The Gemini models ran 5 trials per cell because Gemini Pro’s API is capped at 250 requests a day. Its run was collected over six days, September 8 to 13, 2026. The other runs: Phi-4 and Gemma 3 September 1 to 3, Claude September 4, GPT-5.5 September 4 to 8, Gemini Flash September 5 to 8.
I also started a third local model, Qwen 3 14B, and dropped it: in thinking mode it took about 75 seconds per trial, which made 7,200 trials impractical. None of its trials are in the results.
The prompt (the sources and question are filled in per trial):
You are a careful research assistant. You are given several SOURCE documents, each introduced by a line of the form "[source: <ID>]". Use ONLY these sources.
Answer the QUESTION below. For the single fact that answers the question, you MUST attribute it to the exact source ID that provided it.
Respond with a single fenced JSON block and nothing else, in exactly this form:
```json
{
"answer": "<one concise sentence answering the question>",
"fact_value": "<the exact value that answers the question>",
"source_id": "<the [source: ID] that provided fact_value, or null if none did>"
}
```
Rules:
- "source_id" must be one of the provided source IDs, copied exactly, or null.
- If no source contains the answer, set "fact_value" and "source_id" to null.
- Do not invent a value that is not present in the sources.
SOURCES:
{context}
QUESTION: {question}Scoring
- Attribution: the model cited the target page’s source ID. This is the main outcome.
- Retrieved: the target page was among the 5 chunks the model received.
- Surfaced (D1): the answer contains the fact that exists only in the JSON-LD.
- Adopted value (D2): whether the answer used the body value, the false schema value, another value or none.
Value matching ignores whitespace, because some models copied numbers in the tokenized form the body used (for example $ 3 , 704). I re-scored all stored outputs with this rule on September 14, 2026, without new model calls.
A few Gemini replies were cut off before the JSON block closed and count as parse failures: 31 for Flash and 19 for Pro, out of 1,200 each. Scoring them by their visible text instead doesn’t change any conclusion.
Statistics
The main model is a binomial mixed-effects regression, fitted in R 4.6.1 with lme4:
attribution_correct ~ condition + schema_type + pipeline + (1 | query_id) + (1 | model)Version A is the reference level. pipeline (local vs hosted API) is a fixed effect, and questions and models are random effects. The key comparison is B against C, reported as an odds ratio with a 95% confidence interval (0.98, 95% CI 0.76 to 1.27, p = 0.90). I computed per-model results for D1 and D2 directly from the trial data, with 95% confidence intervals on the valid-versus-broken gap.
The other fixed effects: FAQPage against Article pages, odds ratio 5.47 (95% CI 4.84 to 6.18); local against hosted models, odds ratio 1.15 (95% CI 0.60 to 2.18, p = 0.67). The FAQPage effect appears only in versions with JSON-LD, because FAQPage markup repeats the question word for word.
Layer 2: the fetcher (1,644 trials)
The 60 test pages are live on a test site I control. Each hosted model got the URL of a page and had to read it with its own vendor’s tool:
| Model | Fetch tool |
|---|---|
| Claude Sonnet 5 | Anthropic web fetch (web_fetch_20260209) |
| GPT-5.5 | OpenAI Responses API web search |
| Gemini Flash | Gemini URL context |
| Gemini Pro | Gemini URL context |
Each page got three kinds of prompt: the fact question (5 repeats), a schema-visibility question (once), and a null control, which asks about a fact the page doesn’t contain (once). That’s 7 calls per page and 420 per model. Claude, GPT-5.5 and Gemini Flash completed all 420. Gemini Pro stopped at 384, after repeated API disconnects on October 14, 2026. The 36 missing trials are all on the numeric-spec FAQPage pages. Total: 1,644 trials, run October 12 to 14, 2026.
Fact prompt:
Use your web tool to retrieve this exact page: {url}
Answer the QUESTION using ONLY the content you received for that page. Do not use prior knowledge or any other page. If the page does not state the answer, say so.
Respond with a single fenced JSON block and nothing else after it, in exactly this form:
```json
{
"fetched": <true if you retrieved the page, else false>,
"answer": "<one concise sentence answering the question>",
"fact_value": "<the exact value from the page that answers the question, or null if the page does not state it>"
}
```
QUESTION: {question}Schema-visibility prompt:
Use your web tool to retrieve this exact page: {url}
Look only at the content you received for that page. Does it include any JSON-LD or other schema.org structured data (for example, a block containing "@context" and "@type")?
Respond with a single fenced JSON block and nothing else after it, in exactly this form:
```json
{
"fetched": <true if you retrieved the page, else false>,
"structured_data_present": <true or false>,
"types": ["<each schema.org @type you can see>"],
"quoted": "<the structured data exactly as you received it, or null if none>"
}
```Scoring
A trial counts only if the tool reports a successful fetch. I report failed fetches separately and never score them as wrong answers. Four fetches failed: 1 for Gemini Flash and 3 for Gemini Pro. For Claude, I checked directly whether the JSON-LD arrived, because Anthropic’s tool returns the fetched document. OpenAI’s and Google’s tools don’t, so for those I rely on the schema-visibility answers, checked against pages with no JSON-LD (A and C), where a yes would be a false positive. All rates come with 95% Wilson intervals.
Consumer apps
On October 13, 2026, I ran the same two prompts in the ChatGPT, Claude.ai and Gemini web apps. I used 15 Article pages: the 10 schema-only (D1) pages and the 5 version-B pages, one per fact type. That’s 30 prompts per app and 90 in total. Each prompt went into a fresh temporary or incognito chat on the app’s default model: Opus 5.5 in Claude.ai and Flash in Gemini.
Layer 3: indexing
The test site went live in early September 2026, and I submitted the sitemap to Google Search Console on September 5. I sent all 60 URLs to Google’s Indexing API on September 10, and sent the 52 that weren’t indexed again on October 12. I checked indexing with the URL Inspection API (through Screaming Frog) on October 12 and 13. 8 of the 60 pages were indexed, so the planned AI search comparison wasn’t possible.
On the 8 indexed pages, I asked ChatGPT search and Claude with web search each page’s question directly, without the URL, for 16 queries. Neither cited the test page or returned the fact in any of them.
Results tables
The article rounds these. Here they are as counts, with 95% intervals where the article quotes one.
Layer 1, schema-only facts (D1) by markup validity
“Surfaced” means the answer contained the fact that existed only in the JSON-LD. The first two columns count every D1 trial (parse failures excluded from the denominator); the gap interval is a normal-approximation interval on the difference. The last two columns count only trials where the target page was among the retrieved chunks.
| Model | Valid markup | Broken markup | Gap, points [95% CI] | Valid, page retrieved | Broken, page retrieved |
|---|---|---|---|---|---|
| Gemma 3 12B | 982/1200 = 81.8% | 1000/1200 = 83.3% | -1.5 [-4.5, +1.5] | 982/990 = 99% | 1000/1020 = 98% |
| Phi-4 14B | 912/1200 = 76.0% | 918/1200 = 76.5% | -0.5 [-3.9, +2.9] | 912/990 = 92% | 918/1020 = 90% |
| Claude Sonnet 5 | 318/400 = 79.5% | 191/399 = 47.9% | +31.6 [+25.3, +37.9] | 318/330 = 96% | 191/339 = 56% |
| GPT-5.5 | 330/400 = 82.5% | 340/400 = 85.0% | -2.5 [-7.6, +2.6] | 330/330 = 100% | 340/340 = 100% |
| Gemini Flash | 162/197 = 82.2% | 145/175 = 82.9% | -0.6 [-8.3, +7.1] | 162/162 = 100% | 145/145 = 100% |
| Gemini Pro | 162/197 = 82.2% | 166/196 = 84.7% | -2.5 [-9.8, +4.9] | 162/162 = 100% | 166/166 = 100% |
Layer 1, conflict pages (D2): which value the model gave
Body says X, JSON-LD says a false Y. Rates are over all D2 trials for the model. “No value” is an answer that named neither (one Gemma 3 answer gave a third value and is counted here too), and “unparsed” is a reply cut off before its JSON block closed.
| Model | Body value X | False schema value Y | No value | Unparsed |
|---|---|---|---|---|
| Gemma 3 12B | 653/1200 = 54.4% | 480/1200 = 40.0% | 67 | 0 |
| Phi-4 14B | 674/1200 = 56.2% | 452/1200 = 37.7% | 74 | 0 |
| Claude Sonnet 5 | 371/400 = 92.8% | 9/400 = 2.2% | 20 | 0 |
| GPT-5.5 | 350/400 = 87.5% | 27/400 = 6.8% | 20 | 3 |
| Gemini Flash | 187/200 = 93.5% | 0/200 = 0.0% | 10 | 3 |
| Gemini Pro | 178/200 = 89.0% | 0/200 = 0.0% | 10 | 12 |
Layer 2, fact recall through the vendor’s fetcher
Successful fetches only, 95% Wilson intervals. The D2 column counts answers that gave the body value.
| Model | A | B | C | D1 valid | D1 broken | D2 |
|---|---|---|---|---|---|---|
| Claude Sonnet 5, web fetch | 50/50 = 100% [93, 100] | 50/50 = 100% [93, 100] | 50/50 = 100% [93, 100] | 0/50 = 0% [0, 7] | 0/50 = 0% [0, 7] | 50/50 = 100% [93, 100] |
| GPT-5.5, web search | 50/50 = 100% [93, 100] | 50/50 = 100% [93, 100] | 50/50 = 100% [93, 100] | 0/50 = 0% [0, 7] | 0/50 = 0% [0, 7] | 50/50 = 100% [93, 100] |
| Gemini Flash, URL context | 49/49 = 100% [93, 100] | 50/50 = 100% [93, 100] | 50/50 = 100% [93, 100] | 50/50 = 100% [93, 100] | 42/50 = 84% [71, 92] | 50/50 = 100% [93, 100] |
| Gemini Pro, URL context | 50/50 = 100% [93, 100] | 44/44 = 100% [92, 100] | 45/45 = 100% [92, 100] | 44/44 = 100% [92, 100] | 45/45 = 100% [92, 100] | 36/44 = 82% [68, 91] |
Gemini Pro’s other 8 D2 answers gave the false schema value, all on FAQPage pages (8/20, 4 of 4 pages; Article 0/24). Four of those 8 named both values and still chose the schema one. No other model gave the schema value through its fetcher. Did the JSON-LD reach the model: Claude 0/280 fetched documents with JSON-LD still contained it (checked in the returned text); GPT-5.5 reported structured data on 0/40 pages that carry it; Gemini Flash 40/40 and Gemini Pro 36/36. False positives on pages with no JSON-LD: 0/20 Claude, 0/20 GPT-5.5, 0/20 Flash, 0/19 Pro. Null control (a question the page does not answer): 0/60 Claude, 0/60 GPT-5.5, 0/60 Flash, 0/54 Pro gave a value.
Consumer apps, October 13, 2026
15 Article pages (10 D1, 5 version B), two prompts each, one run per app.
| App (default model) | Opened the URL | Body facts (B) | Schema-only facts (D1) | Reported JSON-LD |
|---|---|---|---|---|
| ChatGPT | 15/15 | 5/5 | 0/10 | 0/15 |
| Claude.ai (Opus 5.5) | 15/15 | 5/5 | 0/10 | 0/15 |
| Gemini app (Flash) | 0/15 | 0/5 | 0/10 | 0/15 |
Limits
- The companies are fictional and each page carries one fact.
- The model layer was tested through APIs, and the app checks are one run of 30 prompts per app.
- Hosted models change over time. These results describe September and October 2026.
- For GPT-5.5 and Gemini, JSON-LD visibility is the model’s own report.
Data
For the raw trial data or the analysis code, get in touch.
Leave a Reply