How LLMs Leak Training Data

Large language models don't just learn from training data. They memorize chunks of it outright, and under the right conditions, they'll hand that data back to a stranger who asks the right way. I've spent enough time in the weeds of extraction research to tell you the tools most enterprises lean on today can't see this problem coming, let alone catch it after the fact.

Nobody sets out to build a model that spits out someone's phone number or a paragraph from a customer contract, word for word. It happens anyway, as a side effect of the basic training objective: predict the next token given everything before it. Sometimes the "pattern" a model learns isn't a pattern at all. It's just a specific sequence it saw often enough to store almost intact.

Researchers split this into a few buckets, and the distinctions matter, because each one carries a different threat.

Verbatim memorization (sometimes called eidetic memorization) is the model reproducing training text word for word, usually from duplication in the training set or overfitting on one sample. Discoverable memorization is a step further and arguably worse: partial or full training samples get reconstructed just through ordinary interaction with the model, no privileged access needed. That beats simply detecting whether a document was used in training. It means the actual content comes back out.

Then there's syntactic memorization. A 2026 study in Nature Communications found memorization gets driven mostly by token overlap, not any real grasp of what the text means. Put plainly: the model can reproduce a paragraph without understanding a single word of it, pattern-matching at the token level while dressed up to look like recall.

So where does this live inside the model? A 2025 study traced it to attention modules in the deeper transformer blocks, while earlier blocks handle the generalization and reasoning work we tend to associate with "intelligence." A model can reason fluently in one part of its architecture while another part sits on verbatim text, waiting for the prompt that unlocks it. Generalization and memorization run in parallel inside the same network. Capability and leakage, in other words, aren't separate problems you can solve one at a time.

Why some training data is far more likely to be leaked than other data

Not all training data carries equal risk. Some of it is inert. Some of it is a live wire. The difference comes down to a handful of measurable factors, and duplication tops the list.

Kandpal et al. (2022) found the relationship between duplication and extraction isn't linear, it's superlinear. A sequence that shows up 10 times in training data gets generated roughly 1,000 times more often than a sequence appearing just once. Risk from duplicated content doesn't creep up gradually. It jumps.

Model size makes this worse, not better. You'd expect a bigger, more capable model to be more sophisticated about generalizing rather than memorizing. Instead, larger models turn out more vulnerable to extraction. Capability and privacy risk move together, which is an uncomfortable pairing for anyone hoping "better" automatically means "safer."

A few other factors compound the picture. Longer prompts raise the odds of triggering memorized content, since they hand the model more context to latch onto a stored sequence. Fine-tuning duration, dataset size, and how similar training samples are to each other all shape how deeply content gets embedded. Models also tend to memorize nouns and numbers first: names, account numbers, phone numbers. The exact tokens functioning as unique identifiers are the ones most likely to stick.

Put this together and you get an uncomfortable conclusion for any enterprise fine-tuning a model on its own data. The conditions that make fine-tuning effective, small datasets, repeated exposure to similar documents, tight domain focus, are the same conditions that deepen memorization. You don't get to keep one without inviting the other.

The concrete types of sensitive data that surface in extraction

This isn't abstract. Researchers have pulled real, identifiable data out of production models, and the categories are worth naming one by one.

Personal information was the first category demonstrated at scale. Carlini et al. (2021) showed GPT-2 could be made to output names, phone numbers, email addresses, and physical locations, recoverable even when the source data appeared in only a single training document. One occurrence was enough.

Copyrighted and proprietary text is another category. Models have reproduced portions of books, news articles, and code verbatim, which stacks a legal exposure problem right on top of the privacy one.

Live credentials should alarm any security team on its own. A scan of roughly 400 terabytes of public web data from Common Crawl's December 2024 archive, covering 2.67 billion web pages, turned up nearly 12,000 live secrets: valid API keys, access tokens, and passwords that still authenticated against services like AWS, Slack, and Mailchimp. Sixty-three percent of those secrets appeared on more than one page. One API key, tied to WalkScore, showed up 57,029 times across 1,871 subdomains. The scanning tool identified 219 distinct types of secrets in that single dataset. This is the kind of data that's gone into widely used AI training corpora. So if a model trained on it learned what a valid Slack token looks like, what else did it pick up the shape of?

Medical data shows the sharpest jump when models get fine-tuned on domain-specific material. Meditron 7B, tuned on clinical guidelines, hit a 30-token memorization ratio of 10.48%, against 1.23% for its baseline. Me-LLaMA, trained on MIMIC-III data, reached 15.32% against a baseline of 0.10%, more than a hundredfold jump. Fine-tuning on sensitive domain data adds retention right alongside capability, and the retention shows up almost exactly where you'd least want it to.

The pattern across all of it: the data categories under the heaviest regulation (health records, credentials, personal identifiers, proprietary IP) are also the categories most likely to show up at high duplication rates inside specialized training sets. Not a coincidence. Same mechanism, playing out across different domains.

How extraction attacks are carried out in practice

Extraction attacks split into two broad categories, and the split tells you a lot about how serious the threat actually is.

Unguided attacks try to pull out any training data at all, no specific target in mind. Targeted attacks work differently: feed the model a prefix, a partial string of text, and try to recover the suffix. Targeted attacks are the more dangerous of the two, because they let someone go after specific information: an email address, a credential, a name tied to a record.

Carlini et al.'s 2021 work on GPT-2 was the founding demonstration here. Just by querying the model with partial text prompts, they recovered hundreds of verbatim training sequences, including PII, IRC chat logs, source code, and UUIDs. What made this unsettling wasn't the volume. The attacks worked even on data appearing just once in training, and it was still recoverable.

The divergence attack, demonstrated by Nasr et al. in 2023, pushed further. Researchers found a way to make ChatGPT break from its normal chatbot behavior and start emitting raw training data, at roughly 150 times its normal rate. Over ten thousand unique memorized examples came out, for about $200 in API queries, a strikingly low cost for that much exposure. The strongest version of their fine-tuning attack extracted training examples from ChatGPT in more than 23% of attempts. Nearly one in four tries succeeded. This also settled something that had been an open question: RLHF, the alignment technique meant to make these models behave, reduces how easy extraction is, but doesn't eliminate the memorization underneath it. The data stays put; alignment just makes it a little harder to reach.

There's a stranger category too: repeated special-character attacks. Certain sequences of unusual characters turn out to be stronger memory triggers than ordinary language prompts. Researchers at Dropbox demonstrated extraction from both GPT-3.5 and GPT-4 using repeated multi-token sequences like this. OpenAI confirmed the vulnerability and issued fixes in early 2024. The fact that it worked at all says something about how thin the line is between generating language and regurgitating it.

When both a base model and its fine-tuned version are accessible, as with DeepSeek-R1 built from DeepSeek-Base, gradient-matching techniques can identify training data that was never released publicly. A narrower but still meaningful risk is membership inference: simply determining whether a specific document was part of the training set, without recovering the text itself. Sounds minor until you sit with it. Confirming a particular patient's record, or a particular company's internal memo, was used to train a model is itself a privacy breach, even without a single word of the original text coming back out.

Why standard measurements understate how much leakage is occurring

Here's where it gets uncomfortable for anyone judging vendor safety off a published benchmark. The metric most commonly used to quantify leakage, Extraction Rate, appears to be undercounting the problem.

A December 2024 study by Tiwari and Suh used sequence-level probability analysis, a finer-grained approach than standard extraction testing, and found Extraction Rate underestimates the actual threat by as much as 2.14 times in randomized LLMs. Not a rounding error: a model that looks reasonably safe on paper might be leaking more than double what the headline number says, because the standard metric misses partial sequence leakages and effects tied to where in a sequence a token sits.

There's a second finding worth sitting with. Experiments show LLMs often generate outputs containing leaked information even when very little of that data existed in training to begin with. And here's the twist: the fewer instances that leaked, the harder that leakage is to detect. Rare leaks hide well precisely because they're rare. You'd expect low-frequency leaks to be low-risk. They're actually the ones most likely to slip past standard testing.

What does this mean for an enterprise reading a vendor's safety documentation? Extraction Rate benchmarks published by a vendor are probably describing a floor, not a ceiling. The real exposure surface is likely bigger than what gets reported, and that's less a story about vendors cutting corners than about the measurement tools the whole field currently leans on.

Where in the AI deployment lifecycle leakage risk appears

Leakage risk doesn't show up at one point. It builds across the whole lifecycle, and each stage adds its own flavor.

Pre-training is the foundation. Public web data gets ingested at scale here, credentials and PII get absorbed passively (nobody's filtering for a stray API key buried in a scraped webpage), and duplication effects lock into the model's weights before anyone downstream even knows they're there.

Fine-tuning is where things get personal, literally, for enterprises. It's the highest-risk stage precisely because this is the point where proprietary documents, customer records, and internal communications become training material. As covered earlier, memorization deepens the longer training runs and the more similar the samples are to each other, which is exactly the profile of most enterprise fine-tuning jobs.

Inference and deployment is where the risk turns visible to actual users, adversarial or not. Prompt-based leakage attacks surface memorized content through what looks like ordinary interaction. Weak output filtering lets sensitive completions slip through to end users who never asked for anything sensitive; they just phrased a question in a way that happened to trigger a memorized sequence.

There's a quieter failure mode too: a model can expose confidential data simply by misreading what a user's asking for, answering an ordinary query, misunderstood, with something it shouldn't have surfaced at all.

Why does this lifecycle view matter so much for enterprises? Because when a company deploys a vendor's LLM, or plugs in a vendor-built agent, it inherits leakage risk from every one of those stages, pre-training, fine-tuning, inference, without having audited a single one of them itself. Every plugin, every connector, every additional fine-tuned agent bolted onto that base model opens a new context where the same memorization dynamics apply all over again. The attack surface doesn't stay fixed. It multiplies with every integration.

What mitigation techniques can and cannot actually guarantee

Real defenses exist here. None of them close the door completely, but some close it further than others.

Deduplication is the best-validated intervention available right now. Models trained on deduplicated data, meaning exact substrings and near-duplicates get stripped before training, show roughly a tenfold reduction in memorized token generation. It targets the amplification mechanism Kandpal et al. identified directly: if duplication is the strongest amplifier of leakage, removing duplication is the most direct countermeasure available.

Differential privacy offers a formal, mathematical guarantee that any single data point's presence in training barely moves the model's output one way or another. Sounds like the ideal fix. It comes with a real cost, though: differential privacy compromises model utility, and it's resource-intensive to apply at the scale modern LLMs run at. The tradeoff between privacy and performance isn't solved here. It's an ongoing negotiation, and right now, performance tends to win.

Machine unlearning is newer: after deployment, can you surgically remove one piece of memorized content without retraining the whole model? Promising in theory. It runs into a chicken-and-egg problem in practice, since you need to know what to unlearn, which means you've already detected it, and detection is exactly the hard part. There's also no standardized benchmark yet for confirming whether unlearning actually worked. It's a tool with real potential and no reliable way, currently, to check whether it's doing its job.

A few narrower techniques round this out. "Goldfish Loss" is a training modification that stops the model from learning random subsets of tokens during training, cutting exact memorization at the source. Output filtering at inference time catches some sensitive completions before they reach a user, but it's bypassable; a slightly reworded prompt often slips past the same filter that caught the original phrasing.

Alignment, RLHF specifically, raises the friction involved in extracting data without removing the data itself. Nasr et al.'s divergence attack proved this directly: an aligned, RLHF-trained model still leaked training data at scale once pushed the right way.

Combine deduplication, differential privacy, and output filtering, and you get a real, measurable drop in risk. Yet no combination of these techniques eliminates memorization outright, and none of them hands you a complete audit trail of what a model actually retained from its training data.

What enterprises cannot see when they deploy vendor AI and why that gap is the real problem

This is the part that matters most to anyone actually responsible for vendor risk inside a company. When you deploy a vendor's LLM, you don't get visibility into what went into it: not the training data composition, not the duplication rates, not the fine-tuning datasets, not what credentials or personal information might have gotten absorbed somewhere along the way.

Traditional security tooling checks application-layer behavior: does the API respond correctly, is access controlled, are inputs sanitized. None of that touches memorization-based leakage. None of it was built to ask whether a model's output is, quietly, a piece of someone's training data walking back out the door.

Generic governance frameworks run into the same wall. Compliance checklists tend to treat "AI risk" as one lump category, but the leakage lifecycle isn't one thing. Pre-training risk, fine-tuning risk, and inference-stage prompt leakage are three different problems calling for three different kinds of scrutiny, and most checklists never draw that line.

The measurement problem compounds all of it. If research-grade Extraction Rate metrics understate leakage by up to 2.14 times, vendor assurances built on those same metrics are offering weaker guarantees than they look like on paper. A vendor citing a low extraction rate isn't lying to you. They're citing a number the research itself suggests is probably too optimistic.

What would actually close this gap? Continuous monitoring of vendor AI, before deployment and all through its use, built to catch emerging extraction patterns, odd outputs, shifts in vendor model behavior over time. A point-in-time vendor risk assessment, the kind most third-party risk programs still run today, can't catch a memorization pattern that only shows up six months after deployment when a fine-tuning update ships quietly on the vendor's side.

That's why tools purpose-built for AI-specific threats, ones combining vendor risk assessment with ongoing detection rather than a general security tool wearing a new label, matter so much right now for TPRM, InfoSec, privacy, and legal teams working this problem together. Even the best tooling only helps a team that already knows what it's looking for. Teams that understand how verbatim extraction works, how a divergence attack functions, how inference-stage leakage slips past ordinary use, are the teams that know which questions to actually put to their vendors, and which answers should make them uneasy.