A Million Tokens of Context Is Still Not a Second Brain

What a Million Tokens Actually Holds

The number is large enough to feel unlimited and small enough to run out. It helps to convert it into objects you already have a sense of.

A token averages roughly three-quarters of an English word, so a million tokens is somewhere near 750,000 words. That is about ten full-length nonfiction books, or a couple of thousand pages. Stated that way it sounds like plenty, and for a single deep task it is: an entire codebase, a long legal matter, a year of one project's documents. The trouble starts when you compare it not to a task but to a life. A person who has taken notes seriously for five years, clipped articles, saved PDFs, kept meeting records, and exported their chat transcripts is well past 750,000 words, often by an order of magnitude. Personal knowledge bases are not book-sized. They are library-sized, and they grow monotonically while the context window stays fixed until the next model.

This is the first structural point, and it does not go away with the next release. Context windows grow in steps, on a vendor's schedule. Your archive grows continuously, on yours. Any architecture that depends on the first outpacing the second is borrowing against a trend line you do not control. Even if a future model holds ten million tokens, the same question returns one archive-decade later, and in the meantime you have built your habits around pasting rather than keeping.

Context Rot: The Part the Benchmarks Buried

The more interesting objection is not capacity but accuracy, and it is well documented. The effect first got a memorable name in the 2023 "Lost in the Middle" work, which found that multi-document question answering follows a U-shaped curve: models are most accurate when the relevant passage sits near the beginning or the end of the context, and measurably worse when it sits in the middle. The finding replicated across model families rather than being a quirk of one vendor.

The follow-on research generalized it into what practitioners now call context rot: output quality declines as input length grows, and it begins declining long before the window is full. Chroma's July 2025 report, "Context Rot: How Increasing Input Tokens Impacts LLM Performance," evaluated eighteen production models, including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3, while deliberately holding task difficulty constant and varying only input length. Its finding is that models do not process context uniformly: performance grows increasingly unreliable as input grows, and it does so even on tasks as undemanding as retrieving or repeating text that is sitting right there. If accuracy slips on the easy cases, the harder ones are not going to hold.

There are architectural reasons rather than mysterious ones. Attention is pairwise, so a hundred thousand tokens implies on the order of ten billion token-pair relationships competing for weight, and any individual passage's share of that attention shrinks as the pile grows. Positional encoding schemes in wide use have a long-term decay property that systematically reduces the computed similarity between distant tokens. And large contexts are full of distractors: passages that are semantically close to your question but not the answer, which is exactly the material an unfiltered personal archive is made of. Ten near-miss versions of the same meeting note do not help a model find the decision you actually made.

Two honest caveats. First, this is an active area and the curves are improving; some recent models handle simple needle-in-a-haystack factoid retrieval at long range with very little position sensitivity. Second, simple retrieval is the easy case. The degradation shows up hardest on multi-hop reasoning, where the answer requires combining facts from several places, which is precisely what you want from a second brain. "Find the sentence containing this phrase" is a search problem that long context handles well. "What did I conclude about this across the last two years, and what changed my mind" is not.

Three Properties a Context Window Does Not Have

Set capacity and accuracy aside. The context window is missing three things that a knowledge base has by definition, and none of them are fixed by making it bigger.

It does not persist

A context window is per-conversation scratch space. When the conversation ends the contents are gone, and the next conversation starts empty. Vendors have layered memory features on top precisely because the window itself does not remember, but those features are summaries and extracted facts, not your documents, and they are a separate system with separate limits. Nothing accumulates in a context window. A library is defined by accumulation.

It is not inspectable or correctable

If a model draws the wrong conclusion from a pile of pasted material, you have very little purchase on why. You cannot open the note that misled it, see that it was a draft you abandoned, and delete it. You re-paste and hope. A knowledge base gives you the thing itself: a file, with a date, that you can read, correct, or throw away. The difference matters most in exactly the situations where being right matters most.

You pay for it every single time

Input tokens are billed per request. Astra's published API pricing starts at ten dollars per million input tokens, so the naive pattern of pasting your entire library into every conversation costs roughly ten dollars per question, before the model writes a word of output. Prompt caching softens this for repeated identical prefixes and is genuinely useful, but the shape holds: re-sending everything is a per-query cost, while selecting the right ten thousand tokens is close to free. The economics point the same direction as the accuracy curves, which is usually a sign the underlying reason is structural rather than incidental.

Working Memory, Not Long-Term Memory

The cleanest way to think about this is to stop treating the context window as storage and start treating it as attention. It is the model's working memory: the material currently under consideration, assembled fresh for one task, discarded afterwards. Human working memory is famously small and that is not a defect, because it sits on top of a much larger store and pulls from it as needed. The pairing is the design. Neither half works alone.

Under that framing the million-token window is a genuine and large improvement, and it changes what is worth doing. It means the working set can be an entire project rather than a handful of excerpts. It means you can stop aggressively chunking documents into fragments that lose their context, because whole documents fit. It means the retrieval step can afford to be generous and imprecise, pulling fifty candidate notes instead of five, since the model can hold them all and sort it out. Every one of those is a real gain. None of them is the same as not needing a library.

The Division of Labor That Actually Works

Once the two are separated the workflow is obvious, and it is the one that quietly won across 2026 rather than any of the more dramatic predictions. Keep everything in a durable store you control. Let a retrieval step choose what is relevant to the question at hand. Hand that selection, generously sized now that windows are large, to the model as context. Read the answer knowing which sources it came from, because the retrieval step can tell you.

The debate framed as "long context versus retrieval" was always a false binary. Long context made retrieval easier by raising the ceiling on how much it can pass along and lowering the penalty for imprecision. What it did not do is remove the need to have the material somewhere addressable in the first place, kept in a form you can inspect, and selected rather than dumped. The selection step is the part that stays valuable, and it is the part a bigger window makes more forgiving rather than unnecessary.

This is also why the practical advice has not changed much even as the models have changed enormously. Keep your notes where you can read them. Prefer formats that outlive applications. Let something do the organizing so the store stays usable as it grows, because an archive nobody can navigate is functionally the same as no archive. Then point whatever model you like at it. The model is the part you should expect to replace every few months; the library is the part you should expect to keep.

This is the design Mindly is built around: everything lands in one local library on your Mac, gets tagged, summarized, and connected on arrival, and stays plain and portable so the next model can read it too. How the library works →

What to Do With a Very Large Window

Practical adjustments worth making now that a million tokens is available, none of which involve abandoning your notes.

• Stop chunking so aggressively. The old habit of splitting documents into small fragments existed to fit tight windows and it routinely severed the context a passage needed. Whole documents fit now; pass them whole.
• Widen the retrieval net, then let the model narrow it. Pulling forty plausibly relevant notes and asking the model to identify the three that matter is now cheap and works well. Precision at the retrieval step matters less than it did.
• Keep the working set scoped to one question anyway. Not because it will not fit, but because accuracy falls as the pile grows and distractors are the main cause. A focused hundred thousand tokens beats an unfocused eight hundred thousand.
• Put the most important material at the beginning or the end. The U-shaped curve is a real effect with a free mitigation: position matters, so place your key sources where attention is strongest.
• Never treat a chat as the record. Whatever you conclude in a long-context session, write the conclusion somewhere durable. The window is emptied when you close the tab.

The Honest Version

Million-token context is one of the more useful capability jumps of the last few years and it deserves the attention it got. It genuinely retires a category of engineering busywork, and anyone who spent 2024 building elaborate chunking pipelines has earned the right to delete some code. The claim that does not hold is the broader one, that organizing your knowledge was a workaround for small windows and can now be skipped.

Organizing was never about the window. It was about being able to find what you know, check where it came from, correct it when it is wrong, and still have it in five years when the current model is a footnote. A context window does not do any of those things, at any size. It was not built to. Give it a good working set and it is remarkable. Ask it to be your memory and you will get something that is confident, expensive, and quietly unable to tell you why.