What the Research Found
Fuzzy duplicates, text modified up to 10 percent, retain 60 to 80 percent of the memorization impact of exact copies.
The Imperial College study, led by Yves-Alexandre de Montjoye with co-leads Matthieu Meeus and Igor Shilov, examined how language models memorize training data. The standard practice in the field is exact deduplication: removing sequences of tokens that appear multiple times verbatim. The premise is that memorization happens when the model sees the same text repeatedly, so removing repeats should prevent it.
The researchers tested what happens with fuzzy duplicates, sequences that share a core but differ in wording. They found that these fuzzy matches slip through standard deduplication and still contribute heavily to memorization. At 10 percent modification, where one in ten tokens differs, fuzzy duplicates retained 60 to 80 percent of the memorization impact of exact duplicates. The safeguard was not working.
The scale of the problem is what makes it significant. For every thousand exact duplicate sequences in the training data, the researchers identified over twenty thousand fuzzy duplicates that escaped detection. The attack surface is not a narrow edge case. It is twenty times larger than what existing defenses address.
How Mosaic Memory Works
The mechanism the researchers describe is called mosaic memory. The idea is that a model does not need to see the exact same text multiple times to memorize it. Instead, the model combines information from many similar sequences, piecing together a pattern like tiles in a mosaic.
Consider a sensitive document that appears in the training data. Someone paraphrased part of it for a blog post. Someone else quoted a different section in a report. A third person summarized it in their notes. None of these are exact copies. All of them share semantic content with the original. The model sees all of them, and the cumulative exposure builds a representation of the underlying information, even though no single sequence was repeated verbatim.
The earlier research, also from de Montjoye's team, found that memorization is primarily syntactic rather than semantic. The model is not understanding meaning and recalling it; it is pattern-matching on token sequences. Replacing tokens with synonyms disrupts detection but does not disrupt the underlying memorization effect. The information is still in there, just reached through a different path.
This is why the problem is hard. Semantic similarity is what makes fuzzy duplicates dangerous, but exact token matching is what makes them detectable. Any filter that catches them has to reason about meaning, which is computationally expensive and itself requires AI models to do well. The defense has the same limitations as the attack.
What This Means for Enterprise AI
If your company uses AI on sensitive documents, the deduplication in the training pipeline is not protecting you.
Enterprise AI deployments often rest on an assumption: data processed by the model is not retained for training, or if it is retained, deduplication ensures it cannot be memorized and extracted. The first assumption depends on vendor policy. The second assumption is what this research undermines.
Consider an enterprise that uses an AI tool across many employees. Each employee submits queries that include fragments of internal documents. No single query contains a complete document. But across hundreds of employees, the aggregate submissions cover the document many times over, in overlapping paraphrased fragments. If the vendor uses that data for training, the fuzzy duplicates accumulate, and the model develops a representation of documents that were never submitted whole.
The Imperial researchers explicitly note that consumer-facing AI tools may not adequately stop sensitive information from being retained across similar interactions. They are talking about the scenario above. The safeguards that vendors describe, the ones that sound reassuring in privacy policies, are the safeguards the research found do not work.
This does not mean every AI tool is leaking your data. It means the guarantee you thought you had is weaker than advertised. The right response is not panic. It is recalibration: understand that deduplication is not a complete defense, and factor that into what you trust AI with.
The Personal Knowledge Risk
The same dynamic applies to personal use. If you write in an AI assistant about something sensitive, and similar content has appeared elsewhere in training data, the model may retain patterns that could reconstruct what you wrote. You are not the only person with a medical question, a legal concern, a financial situation, a personal secret. The model has seen thousands of similar queries. Yours adds to the mosaic.
The practical risk depends on what you are writing and who is trying to extract it. Random extraction from a giant model is statistically unlikely. Targeted extraction, where an adversary knows roughly what they are looking for and can craft prompts to surface it, is a different threat model. The research demonstrates that the information can be in there; how hard it is to get out depends on context.
The healthiest assumption is that anything you type into an AI assistant could theoretically be reconstructed by someone with access to the model and sufficient determination. For most people and most queries, this risk is low enough to ignore. For genuinely sensitive information, it is a reason to use a local model that does not train on your data, or to not use AI at all.
A local-first second brain keeps your sensitive notes on your device, where they are not part of anyone's training data. Local-first notes →
What Vendors Should Be Doing
The research points to several directions for improvement. Fuzzy-aware deduplication, which looks for semantic similarity rather than exact matches, can catch the duplicates that current filters miss. The tradeoff is computational cost: comparing every sequence to every other sequence for semantic similarity is expensive at training-data scale.
Differential privacy techniques can bound how much any individual data point influences the model. Machine unlearning can attempt to remove specific information after training, though current techniques are imperfect. The field is active; the solutions are not yet deployed at scale.
What vendors should not be doing is claiming that existing safeguards solve the problem. The Imperial research is public and replicable. Any vendor citing deduplication as sufficient protection for sensitive data is either unaware of the research or hoping you are.
The honest answer from vendors would be: we reduce memorization risk through deduplication and other techniques, but we cannot guarantee that no sensitive information is retained. Some vendors are moving toward this framing. Others are not. The difference tells you something about how seriously they take the problem.
What to Do With This Information
- Understand that deduplication is not a complete defense. The safeguards vendors describe reduce memorization risk but do not eliminate it. Calibrate your trust accordingly.
- Treat AI assistants as you would a colleague who takes notes. Do not share information you would be uncomfortable seeing in someone else's notes, even if accidentally.
- Use local models for genuinely sensitive work. If the information must not reach anyone else's servers, do not send it. Local inference is now good enough for many tasks.
- Read privacy policies for what they actually say. Look for language about training, retention, and data use. If a vendor claims deduplication solves memorization, they are not being fully honest about the research.
- Separate your archives from your AI interactions. Keep sensitive notes in a system you control. Use AI to help with them via local models or read-only access, not by uploading everything to a hosted service.