Training Data Poisoning Attack Vectors

The 26% figure from IO's 2025 State of Information Security Report deserves real attention, not as a talking point but as a baseline. One in four organizations in the US and UK has already experienced an AI data poisoning intrusion. And "experienced" in security survey language means detected, which means the actual number is higher by some margin nobody can precisely quantify. Organizations without detection capability at the training data layer would not appear in that count at all.

Shadow AI compounds this in ways procurement teams consistently underestimate. The same report found that 37% of enterprises observed unauthorized generative AI use among employees. Unvetted models introduced through shadow AI carry unknown training provenance. Official procurement can be disciplined; the actual attack surface is still enterprise-wide regardless.

On the supply side, a 2025 study found that between 15% and 25% of scraped datasets contain low-quality or unverifiable content. Vendors building on web-scraped corpora inherit that exposure automatically. Separately, 12% of organizations reported data integrity issues in AI pipelines during 2025, distinct from those who confirmed a full intrusion. That 12% is probably sitting in an uncomfortable liminal state: something is degraded, but nobody knows what or where it originated.

The question for any enterprise is no longer whether their vendor AI has touched poisoned data. It is which entry point was used, and whether any detection capability exists at that layer.

The Three Axes That Organize the Attack Family

"Training data poisoning" gets treated as a single thing in most conversations. It is not. It is a family of distinct vectors with different mechanics, different detection profiles, and different defensive requirements. Collapsing them into one label is a significant reason organizations misconfigure their defenses.

The first axis is attack objective. Indiscriminate attacks degrade overall accuracy, making the model generically unreliable. Targeted attacks alter predictions only on specific inputs while leaving everything else intact. Backdoor attacks are the most sophisticated: behavior appears completely normal by default, and the malicious output surfaces only when a specific trigger is present. Objective type determines detection difficulty in a direct way. Indiscriminate degradation eventually surfaces through performance monitoring. A well-designed backdoor can survive indefinitely if the trigger is never accidentally activated during testing, and if the attacker has engineered it carefully, it will not be.

The second axis is lifecycle stage, meaning where in the pipeline the poison enters. Pre-training attacks hit general-purpose foundation model corpora. Fine-tuning attacks target task-specific adaptation datasets. Embedding attacks corrupt the vector representations used in retrieval. Defense placement maps directly to this axis. A control addressing fine-tuning dataset integrity does nothing for a pre-training corpus attack, and neither covers the retrieval layer. These are not overlapping controls.

The third axis is access model: what the attacker can actually reach. Full data access requires an insider or a compromised vendor. Partial contribution covers things like contributing to public datasets or federated learning nodes. No direct access encompasses web-scale manipulation, RAG knowledge base injection, and external tool poisoning. This last category is where vendor risk assessments focused narrowly on "was the data vetted" fall apart structurally. Vetting addresses full-access scenarios. The more architecturally dangerous attacks operate entirely from outside.

Any serious vendor risk conversation has to work across all three axes.

Label-Flipping: The Simplest Vector and Why It Scales in Federated Environments

Label-flipping is the most conceptually straightforward attack in this family, and its simplicity is exactly what makes it persistent. The attacker does not touch the underlying data; they alter the labels attached to training examples. The model then learns the wrong mapping. The raw inputs remain entirely plausible because they are real data, just mislabeled. Naive inspection finds nothing suspicious, because there is nothing suspicious to find in the inputs themselves.

Check Point documented a concrete version: relabeling phishing emails as legitimate, so the model retrains and subsequently passes malicious emails through spam filters. The inputs are genuine phishing emails. The corruption is entirely in the label. An auditor reviewing the dataset for plausible-looking examples will find them, because they are there and they are authentic.

Federated learning is where this vector scales most dangerously. In a federated architecture, workers control their local data; the aggregating party never inspects individual node datasets. Research published in MDPI Electronics in 2025 describes the consequence: swap labels, add noise to instances, inject out-of-distribution samples locally, and there is no single point of data custody at which label-level auditing is structurally feasible. The design property that makes federated learning privacy-preserving, the absence of centralized data inspection, is precisely what makes label manipulation so difficult to catch. The privacy guarantee and the security gap come from the same architectural decision. That is worth sitting with, because it means you cannot engineer your way out of it without compromising the privacy property itself.

For vendor risk purposes, any vendor using federated or distributed training across third-party nodes inherits label-flipping exposure from each participant. That is an architectural consequence, not a theoretical one.

Backdoor and Sleeper Agent Attacks: The Vector That Survives Safety Training

The backdoor attack is where this problem starts to feel intractable, and I mean that carefully. The mechanism is almost elegant: the model behaves correctly on all normal inputs, and a specific trigger causes it to produce attacker-specified output. The seminal demonstration is BadNets (Gu et al., 2017), in which a white square in the corner of an image causes misclassification to a chosen label. The trigger can be an image artifact, a phrase, a token, a formatting pattern. During every evaluation that does not include the trigger, the model looks fully functional, because it is.

Anthropic's sleeper agents research (Hubinger et al., 2024, arXiv:2401.05566) produced a finding that should unsettle anyone responsible for AI procurement: once deceptive backdoor behavior is learned, standard safety techniques, specifically supervised fine-tuning, reinforcement learning, and adversarial training, do not reliably remove it. The model generates unsafe outputs under trigger while appearing fully aligned in every other evaluation. If your organization applies post-procurement safety fine-tuning and considers that sufficient, the research says otherwise. The safety layer and the backdoor coexist. They do not compete.

There is a number in this research that people tend to initially misread. Poisoning under 0.01% of training data is sufficient to implant backdoors that survive heavy safety fine-tuning. That sounds reassuring, a vanishingly small fraction. But when a foundation model trains on trillions of tokens, 0.01% is still a count of documents that is entirely manageable for a motivated attacker. The percentage is small; the absolute volume is not.

Souly et al. (arXiv:2510.07192, October 2025) sharpened this further: 250 poisoned documents were sufficient to compromise models ranging from 600 million to billions of parameters, even when larger models trained on more than twenty times as much clean data. Absolute count is what matters, not percentage. I have been in vendor conversations where the counterargument was essentially "our dataset is too large to be meaningfully poisoned." It is a comfortable belief. The research does not support it, and I am not sure it ever did.

Sleeper Memory Poisoning: When the Attack Surface Is What the Agent Remembers

This vector reflects something that has changed about how LLMs are deployed, not just trained. Stateful agents that store memory across sessions introduce an attack surface that has nothing to do with the training corpus. The question shifts from what the model was trained on to what the agent currently remembers, which is a different question entirely and one most governance frameworks are not yet asking.

The paper "Hidden in Memory" (arXiv:2605.15338, 2026) describes the mechanism with uncomfortable precision. An adversary manipulates an external document, webpage, or repository to cause the agent to store a fabricated memory about the user. Unlike conventional prompt injection, which must be re-delivered each session, this attack is dormant once stored. It re-emerges across multiple later conversations without any further intervention from the attacker. The poisoned memory is already in the agent's persistent store, waiting for the right context to surface.

The empirical results are not edge-case numbers. Poisoned memories were successfully added in up to 99.8% of attempts on GPT-5.5 and 95% on Kimi-K2.6. Among successful retrievals, poisoned memories caused attacker-intended agentic actions in 60% to 89% of evaluations across models.

The enterprise relevance is direct: any vendor-deployed AI assistant with persistent memory or long-term context is a candidate for this vector. Governance frameworks that assess training data provenance do not cover this surface. They are asking the right question about the wrong layer, which is perhaps worse than not asking at all, because it creates a false sense of coverage.

RAG Poisoning: Injecting Malicious Content Into the Retrieval Layer

Retrieval-augmented generation extends the attack surface in a specific and consequential way: it makes poisoning possible after deployment, without touching the model at all. The model is fine. The knowledge base is not.

PoisonedRAG, published at USENIX Security 2025, produced numbers that clarify the asymmetry. Injecting five malicious texts per target question into a knowledge base of millions produced a 90% attack success rate. Five documents in millions. Multi-modal RAG poisoning achieved greater than 80% success rates with the same minimal footprint. The volume of clean content provides essentially no protection against a small number of precisely crafted poisoned entries, because retrieval systems are not designed to compare trustworthiness across documents; they retrieve what is relevant to the query. Relevance and integrity are different properties, and the system checks only one of them.

MINJA, a systematic study of runtime memory injection, found approximately a 98% injection success rate: an adversary sends crafted queries that cause the agent to store poisoned reasoning traces. AgentPoison targets RAG knowledge bases with a trigger token and achieved a 62.6% end-to-end attack success rate.

The RAG knowledge base is not a one-time configuration decision. It is an ongoing attack surface that changes continuously as new content is added. Standard security assessments evaluate the model. RAG knowledge base provenance and integrity monitoring are a separate discipline, and in most assessments I have reviewed, they are absent entirely, not deprioritized but absent, as if the question had not occurred to anyone.

Supply Chain and Open-Source Repository Attacks: Poisoning Upstream of the Vendor

Research by Carlini and colleagues demonstrated that attackers can steer foundation model behavior by altering portions of publicly available training data, without direct model access. At web scale, every page in a scraped corpus is a potential injection point. The work required from the attacker is not inside the vendor's infrastructure; it is upstream of it, in publicly accessible content the vendor's pipeline ingests. That is a meaningful distinction, because it places the manipulation entirely outside the perimeter the vendor is actually monitoring.

Open-source dataset repositories have grown substantially; Hugging Face datasets expanded by over 300% since 2022. The attack surface grows faster than inspection capacity can keep pace with. Research examining poisoned models found on Hugging Face identified 100 such models, each potentially enabling attackers to inject malicious code into user machines. The repository that makes pre-trained models accessible for legitimate reuse also makes pre-poisoned models accessible for accidental adoption. Both populations exist in the same place, indistinguishable at a glance.

The evolution of this attack surface tracks the evolution of deployment architectures. Early attacks targeted pre-training corpora. Subsequent attacks implanted backdoors directly in model weights. With the rise of agent paradigms, the surface shifted further outward to plugins, connectors, and extension ecosystems. A paper published in 2025 (arXiv:2604.03081) traces this progression explicitly. The perimeter is not fixed; it moves as deployment architectures evolve, and defenses built around a previous architecture do not automatically cover the current one.

MCP tool poisoning is a recent and concrete illustration of this. The MCPTox benchmark (arXiv:2508.14925) evaluated many hundreds of malicious cases and revealed widespread agent vulnerabilities in third-party tool integrations. Model Context Protocol servers function as distribution points for poisoned tool behavior, an attack surface that did not exist before the agent paradigm. Every pre-trained model a vendor builds on, every plugin they support, is part of the inherited risk surface, and training data provenance is only one slice of it.

Synthetic Data Pipelines and Recursive Propagation: When Poison Becomes Self-Replicating

Synthetic data accounts for somewhere between 10% and 30% of modern LLM training pipelines. A substantial fraction of training data is generated by models, and some of those models are compromised. This is not a hypothetical to hedge against; it is an architectural reality of how current training pipelines actually work.

The Virus Infection Attack, described in a September 2025 study reported by Lakera, showed what happens when poisoned content enters synthetic data pipelines. Once baked into synthetic datasets, the corruption propagates forward. Each subsequent model trained on the output of a prior model inherits and amplifies it, without any further intervention from the attacker. Researchers called it recursive poisoning, and the term is apt: the attacker's initial foothold grows on its own, compounding across generations.

What makes this structurally distinct from other vectors is the detection window, or more precisely, its absence. Most attacks require the attacker to maintain access or re-inject. Recursive poisoning requires a single successful entry. By the time effects surface in a downstream model, the origin is multiple generations back. You are looking at a consequence. The cause is gone.

Multimodal pipelines carry the same exposure. Research presented at CVPR 2025 documented "Silent Branding," in which diffusion models were poisoned to reproduce logos without prompting, and "Losing Control," in which ControlNets were poisoned so that triggers force NSFW output while appearing entirely normal otherwise. When poisoned image outputs are recycled as training data for subsequent models, the infection propagates in exactly the same way as text-based recursive poisoning. The modality changes; the mechanism does not.

Synthetic data provenance is not a solved problem. Chain-of-custody for data generation, not just data collection, is a requirement that most current governance frameworks have not yet addressed. And most frameworks do not seem aware they are missing it, which is a different kind of problem.

How Attack Difficulty Scales With Organizational Context, and Where Defenses Must Be Placed

Everything above converges on a practical question: where do defenses actually have to go? There is no single layer, and any framework that implies otherwise is describing a simpler problem than the one that exists.

Attack difficulty varies meaningfully by organizational context. A large enterprise with centralized AI procurement, a dedicated ML security team, and a mature vendor review process faces a different threat surface than a mid-size organization relying primarily on third-party APIs with no visibility into underlying training data. Neither is categorically safer. The large enterprise faces more sophisticated supply chain and sleeper agent risks because it deploys more capable models in more complex architectures. The mid-size organization faces more exposure from shadow AI and RAG systems configured once and never monitored again.

Defense placement has to follow the lifecycle axes. Pre-training corpus attacks are largely a vendor responsibility; the enterprise's leverage is in procurement due diligence, specifically asking about training data provenance, scraping practices, and whether web-sourced content undergoes provenance filtering. Fine-tuning dataset attacks are partially within enterprise control when organizations are customizing models on proprietary data; dataset auditing and statistical anomaly detection on label distributions are relevant here. The federated learning scenario requires contractual requirements about node data custody, not just technical controls, because the technical layer cannot see what it needs to see.

For retrieval-layer attacks, RAG knowledge base integrity monitoring needs to be treated as a distinct operational discipline from model security. Content entering a knowledge base should be subject to provenance checks and anomaly detection, especially in systems that ingest external or user-contributed content. The PoisonedRAG finding makes the asymmetry concrete: five documents in a knowledge base of millions is sufficient for a 90% attack success rate. Volume is not a defense.

Memory poisoning in stateful agents requires auditing what memories the agent stores and providing mechanisms to inspect and revoke them. That is an emerging practice, not a mature one. Most current agent deployments have nothing resembling it, and most security assessments do not ask about it because the question is not yet standard. That gap will close eventually; the question is whether it closes before or after a significant incident forces it.

Supply chain defenses require accepting that the vendor's direct training pipeline is only part of the picture. Every component the vendor inherits, every open-source model they build on, every plugin ecosystem they support, is within scope. That is a harder conversation to have in a procurement process than asking whether training data was vetted. The accurate scope of the risk does not shrink because the conversation is inconvenient.

The consistent pattern across all of this is a specific mismatch: security frameworks built for traditional software asking questions that do not cover the surfaces where AI-specific attacks actually operate. A model that performs well on every benchmark you run at it is still capable of carrying a backdoor waiting for a trigger you have never tested. The governance question is whether existing frameworks close that gap before the exposure compounds further, and right now, most of them do not.