Consistency Is Not Completeness
The standard benchmarks for summarization quality measure the wrong thing, and they do it well enough that the right thing looks solved.
Most summarization evaluation asks whether the summary contradicts the source. This is a sensible thing to measure, and modern systems pass it handily. The trouble is that a summary can be perfectly consistent with a source while omitting the only part of the source you care about. Consistency is a floor, not a ceiling. A one-sentence summary of a ten-thousand-word contract that says 'this is a legal agreement between two parties' is not false. It is also not useful for deciding whether to sign.
Completeness, by contrast, asks whether the summary preserves the information needed for a specific downstream task. The task might be making a decision, answering a question, finding a reference, or reconstructing an argument. Different tasks need different facts from the same source, which means completeness is not a property of the summary alone. It is a property of the summary relative to what you will do with it, and you often do not know what that is at the time of compression.
This is the structural problem, and it is why the validity question does not go away with better models. Compression is lossy by definition. A summary shorter than its source contains less information than its source. The question is not whether information was lost, but whether the lost information was the kind you could afford to lose. And that question cannot be answered until you know what you needed, which is usually later.
What the Measured Studies Found
The June 2026 paper 'When Summaries Distort Decisions' tested this directly. Researchers took financial filings and earnings-call transcripts, compressed them with production LLMs at a 90% reduction ratio, and measured whether the compressed version supported the same investment forecast as the original. Naive compression, the kind most pipelines run, produced decision flips in roughly 40% of cases. The summaries were fluent and factually plausible. They simply no longer contained the signals the original contained.
The effect is not limited to finance. A 2025 review of 12,999 clinician-annotated sentences from LLM-generated clinical notes found a sentence-level hallucination rate of 1.47%, which sounds reassuring until you learn that 44% of those hallucinations were rated major, meaning they affected diagnosis or management. In medical case summaries specifically, hallucination rates reached 64% without mitigation prompts, dropping to 23% with structured interventions. The ceiling is high and the floor is low, depending on the task and the safeguards.
A separate line of work documents the positional problem: models summarize beginnings and endings more faithfully than middles. A 2024 study across six summarization datasets found faithfulness at document start around 90% but mid-document faithfulness dropping to 65%. This is the same U-shaped curve that affects retrieval in long contexts, arriving from a different direction. If the critical paragraph is on page seven of twelve, the summary is more likely to omit it than if it appeared on page one.
| Study | Finding | What it means |
|---|---|---|
| Financial summarization (2026) | 40% decision flips at 90% compression | Summaries that look faithful can reverse what you would conclude |
| Clinical notes review (2025) | 1.47% hallucination, 44% major | Low rates still mean high stakes when errors are serious |
| Positional faithfulness (2024) | 65% mid-document vs 90% start | Models compress unevenly, favoring beginnings and endings |
Where Summarization Lives in Your Stack
Summarization is not only a user-facing feature. It is load-bearing infrastructure in most AI-augmented workflows, and the losses compound.
Chat history compression is the most common instance. As conversations grow beyond context limits, systems summarize older segments to free up tokens. At a 10:1 compression ratio this works beautifully, until the user references a constraint they mentioned twelve turns ago, one that got compressed into irrelevance. The system has no way to know the detail mattered until the question arrives, and by then the detail is gone.
Retrieval-augmented generation is another. RAG pipelines often summarize retrieved chunks before passing them to the generator, either to save tokens or to remove noise. The summarization step can remove exactly the sentence the generator needed. You notice this as a wrong answer, not as a missing input, which makes it hard to debug.
Automatic note summarization is a third. The promise is that you capture everything and the system distills it. The risk is that distillation loses the specific reference, the exact phrasing, the numerical detail, and what remains is a fluent gloss on something you no longer have. If the original is still available the summary is fine. If the summary replaced the original, or if the original is too long to ever revisit, the loss is permanent.
The compounding problem is that summaries often feed other summaries. A summary of a meeting becomes an input to a project summary, which becomes an input to a quarterly rollup. Each step is lossy, and the losses are not random, they are systematically biased toward what is easy to compress, which is usually not what is specific or surprising or the reason you would go back.
Why the Problem Does Not Go Away
The tempting assumption is that better models will solve this. They will not, for a structural reason. A summary shorter than its source must lose something, because information theory says so. The question is which something, and that question has no task-independent answer. A summary optimized for one use, catching up quickly, will drop details needed for another use, verifying a specific claim. No single summary serves all purposes, and most systems produce a single summary.
Better models do reduce hallucination, the fabrication of content not in the source. That is real progress and worth having. But faithfulness to the source is a separate axis from completeness for the task, and gains on the first do not automatically produce gains on the second. A model that never invents facts can still routinely omit them.
The practical implication is that summarization is a tool, not a replacement. It is useful for skimming, for getting oriented, for deciding whether to read the full thing. It is not reliable for preserving every detail you might later need, because the system cannot know which details those are and neither can you. The workflow that survives is one where the summary points at the source rather than standing in for it.
What to Do About It
You are not going to stop using AI summarization, and you should not. The adjustments are about where you place your trust.
- Keep the source. A summary is an index, not an archive. If the underlying document matters, store it where you can find it. The summary can help you decide whether to open the source; it should not be the only thing you have.
- Verify specifics. If a summary surfaces a number, a name, a date, a commitment, check it against the original before acting. The kinds of details that matter most are exactly the kinds most likely to be dropped or softened.
- Treat summaries of summaries with extra suspicion. Compounding losses are real. A rollup that summarizes a summary that summarized a meeting note has passed through three lossy stages, and what remains may be smooth prose with no grounding.
- Use structured extraction where structure matters. If you need specific fields from a document, ask for those fields explicitly rather than asking for a summary. The summary is what the model thinks is important; the fields are what you know you need.
- Test your pipeline on cases where you know the answer. The only way to know whether your summarization workflow preserves decision-relevant information is to run it on material where you can check. Most people never do this, and it is worth the hour it takes.
Mindly keeps the original alongside any summary, because we watched people lose exactly this way. How it works →
The Honest Trade-off
Summarization is one of the most useful things AI does for knowledge work. It turns an overwhelming pile of material into something you can skim in minutes. It is not going away and it should not. The point is not to avoid it but to use it for what it is good at, which is orientation, not preservation.
The risk is not that summaries lie. Modern systems rarely do. The risk is that they are incomplete in ways you cannot see until the missing piece is exactly what you needed. That risk is managed by keeping the source accessible, by verifying before acting, and by treating the summary as a pointer rather than a record. Do that and summarization is a massive time-saver. Skip it and you are trusting a compression algorithm to know what you will care about six months from now, which is a bet that has been measured and does not pay.