On 20 August I nearly published a list of fourteen companies. The press release I believed I was citing names seventeen.
I was drafting an answer page on x402, the payment protocol, and I asked a web search who the premier members of the x402 Foundation were. The summary came back fluent: fourteen names, attributed to the Linux Foundation's launch announcement. It read like a fact with the source already attached. Then I fetched the announcement itself and counted. Seventeen. The summary had dropped Monad Foundation, Solana Foundation and Stellar Development Foundation from the back half of an alphabetical list, and nothing in its answer showed a seam.
A wrong list with a citation attached is worse than no list. If the search had failed I would have gone straight to the press release. Because it answered, and answered politely, fetching the source felt like ceremony. It was the only step that mattered.
An invented table at a dead URL
Five days later, drafting a page on who is liable when an AI agent buys the wrong thing, I asked what the AP2 protocol says about liability. The summary told me the AP2 specification carries a liability-allocation table, quoted a sentence about every payment network defining its own liability contracts, and handed me the specification's URL. That URL was a 404. The live specification sits one path segment away, and the word liability appears on it zero times. There is no table on that page and no such sentence anywhere on it.
The summary was not inventing from nothing, which is what makes this one hard to catch. AP2's FAQ really does address the subject, in different words: the protocol's stated objective is to provide evidence that helps payment networks establish accountability and liability principles. The claim existed, and the summary moved it to a different document, hardened prose into a table, and minted a quotation. Each step made it easier to cite.
When the vendor's own pages disagree
The third case is quieter. The same x402 page needed the date AWS launched AgentCore Payments in preview, and a summary said May. AWS's what's-new item for that launch is filed under April 2026. AWS's general-availability post says the preview arrived in May. The vendor's two pages disagree with each other, and the summary had picked a side without mentioning that sides existed. I cut the date from my page rather than guess. As of today the two AWS pages still carry different months.
The habits of compression
None of this looks like a malfunction, and I do not think it is one. A summary is compression, and compression has habits: it shortens a list from the tail, it merges neighbouring documents into one, and it rounds a disagreement into an answer. What comes out is grammatical and sourced-looking, which is a worse failure shape than a blank result, because a blank result sends you to the document.
The lists worry me most, as a practitioner. A wrong name gets caught by the first reader who knows the field, while a dropped name leaves nothing on the page to notice. If the x402 page had shipped naming fourteen premier members, the error would have sat there reading as a complete answer.
What I fetch before I cite
The rule I run now costs about ten minutes a page. Every claim a search hands me gets its underlying document fetched rather than trusted from the snippet. I poll every URL before I link it, because a returned URL is a guess like everything else in the answer. Lists get counted on the primary page, and dates get read in two places. Across my last six site runs that rule is four for four: the shortened member list, the invented table, the dead URL and the April-to-May split were all caught at the fetch, and none of them would have been caught by reading the summary more carefully.
There is a loop in this that I feel while writing the answer pages themselves. The pages I publish exist partly to be read by the same class of machinery that handed me the fourteen-name list: they answer the question in the first 120 words because that is what a summarising engine lifts. I write for the compressor on one side of the desk and defend against it on the other. Both jobs reduce to the same discipline, keeping every number on the page traceable to a document that resolves.
The URL the search returned for the AP2 specification still returns a 404 this afternoon. I polled it again before writing this sentence.