Sensitive Information Disclosure via LLM Responses
The research that made this concrete was a 2021 demonstration by Carlini and colleagues, published at USENIX Security. They ran extraction attacks against GPT-2 and recovered verifiable content: PII, IRC logs, code, UUIDs, strings that had been sitting in the training corpus. What mattered wasn't just that it worked. It was reproducible and systematic, which moved the attack class from hypothetical to documented. But what if the real significance isn't the attack itself, but what it tells us about the systems we're still building?
A follow-on study found three log-linear relationships: memorization scales with model capacity, with how frequently a data point appeared during training, and with the length of the prompt used during extraction. The authors didn't hedge their conclusion. Memorization will worsen as models scale, absent active mitigations. We are building larger models. We are feeding them more data. If scale is simultaneously the goal and the liability, responsible deployment doesn't have an obvious shape yet.
Verbatim memorization is the intuitive form: the model regurgitates exact strings. Semantic memorization is harder to catch. The model reproduces the meaning, structure, or distinctive phrasing of source material without copying it character-for-character, which makes programmatic detection unreliable and still constitutes a genuine privacy exposure.
Research on a large open-weights model found at least one percent of its training corpus was recoverable. One percent against billions of tokens is not a rounding error. Some portion of what's recoverable will be sensitive, and you may be unable to determine which portion.
Fine-tuning sharpens this considerably. Work published in 2025 across GPT-2, Phi-3, and Gemma-2 found that fine-tuning with repeated sensitive data pushed privacy leakage rates from a baseline of roughly zero to five percent up to sixty to seventy-five percent. Enterprises customizing models on proprietary data are operating in a materially different risk environment than those using base models, and if your threat model was calibrated to base model behavior, there's a gap worth finding before someone else finds it for you.
One important wrinkle: a 2026 study found a consistent gap between capability and propensity. Prefix attacks reliably elicit memorization, but in ordinary, non-adversarial use, models rarely surface training data spontaneously. An attacker probing deliberately is a different scenario from a user asking a routine question. Both matter; they require different responses.
The supply-chain dimension is the piece most teams miss entirely. Truffle Security's 2025 analysis of a major training dataset found approximately 12,000 live credentials, including API keys and passwords, with 63 percent appearing across multiple pages. Those secrets entered the dataset through public web content aggregated during corpus construction. No one at the affected organizations did anything wrong. The data was public, the crawl was automated, and the model learned it. When your organization adopts a third-party model, you inherit its memorization profile, with limited visibility into what that profile contains, and the vendor is often in the same position.
Why the context window is the more common attack surface in deployed applications
In a deployed application, the model processes considerably more than your message. Before generating any response, it works through an assembled bundle: the system prompt an engineer wrote to encode business logic and behavioral constraints, documents a retrieval pipeline surfaced from an internal knowledge base, the accumulated conversation history, any outputs returned by tool calls the model already made. That bundle is the context window, and the privacy assumption that breaks down in practice is that it functions as a secure memory store. It doesn't. It's plaintext, and the model can reference any part of it when constructing any output.
What routinely lives in these bundles is not trivial. System prompts sometimes contain operational credentials or safety mechanisms the developer wants kept confidential. RAG pipelines surface internal wikis, contracts, HR records, financial documents. Conversation history accumulates whatever sensitive information a user typed across prior turns. Tool call outputs can include raw API responses and database query results.
Research has shown that extraction from inference-time context is not theoretical. Work by Panda and colleagues in 2024 demonstrated a targeted neural phishing approach that achieved success rates up to 50 percent in tricking a model into disclosing PII from its context. In a hardened model, a coin-flip success rate is nowhere near immunity. That raises an important question: if a hardened model still leaks at that rate, what does that imply about models that haven't been hardened at all?
OWASP's LLM07:2025 entry on System Prompt Leakage makes this explicit: confidential system prompts, including the operational secrets and the security mechanisms themselves, can be extracted through targeted prompting. This is an inference-time attack, rather than a training-data attack, and that distinction reshapes where the defensive work needs to happen. You cannot solve it by adjusting how the model was trained. The data wasn't in the training corpus; it was passed to the model at runtime. The relevant controls are what you include in the context bundle, what permissions govern what gets retrieved, and how carefully you audit system prompt contents before deployment.
Prompt injection as the technique that weaponizes both memorization and context leakage
Prompt injection is adversary-crafted input that causes a model to abandon its intended instructions and execute something else. OWASP ranks it first in the LLM Top 10 for 2025, and the reasoning isn't complicated: it is the primary trigger mechanism by which the disclosure risks above get actively exploited.
The architectural root cause is not a bug in any particular implementation. LLMs cannot reliably distinguish between trusted system instructions and untrusted user or external content. The same attention mechanism that makes them generalize, follow instructions, and reason across context also makes them susceptible to instructions embedded in data they process. The UK's National Cyber Security Centre noted in December 2025 that prompt injection is a problem that is never fully fixed, for exactly this reason. It is structural, not incidental.
The scale is considerable. A 2025 study cited by Proofpoint documented over 461,640 prompt injection submissions in a single dataset, with success rates ranging from 50 to 84 percent depending on technique. As of late 2024, roughly 90 percent of successful prompt injection attacks resulted in sensitive data leakage. The injection is rarely the end goal; the leak is.
Direct injection is the simpler form: an attacker controls the user-facing input and crafts the malicious prompt themselves. Indirect injection is more dangerous in production. The attacker plants instructions in content the model will retrieve, a document in a knowledge base, an email an AI assistant will process, a webpage a browsing-enabled model will visit. The model encounters the embedded instruction during retrieval and executes it on behalf of whoever triggered the retrieval. The attacker never needs direct access to the system.
This is why treating disclosure risk and injection risk as separate concerns, addressed by separate teams with separate remediation plans, creates a gap. Output filtering and confidentiality directives in the system prompt are real controls. An injected prompt can override them entirely. Patching the output layer without addressing the injection vector means the security mechanism itself becomes circumventable. But how does this affect our original promise of defense-in-depth? It suggests that layered controls are only as strong as the layer an attacker chooses to target first.
How RAG pipelines introduce a third disclosure pathway that most teams underestimate
Retrieval-augmented generation has become the default architecture for connecting LLMs to internal knowledge. Rather than fine-tuning a model on proprietary data, which is expensive, slow, and carries its own risks, organizations build a retrieval layer that fetches relevant documents at query time and passes them into the context window. Per OWASP's 2025 data, 53 percent of companies deploy RAG pipelines. This is not a niche architecture.
The core disclosure risk follows directly from how retrieval works. If the pipeline's retrieval permissions are broader than the querying user's authorization, the model will surface and summarize documents the user was never supposed to see. The LLM becomes, in effect, a search-and-summarize interface for data the user couldn't access through any direct channel. The access control failure happens at retrieval, not at the model layer, so output filtering cannot remediate it. You cannot redact what was already retrieved.
The categories of data that RAG systems routinely index are precisely the categories regulators care about: PII, credentials, financial records, legal files, health information, strategic plans. These are also the contents organizations are most motivated to make available to AI assistants, which is the actual tension, and their unauthorized disclosure creates the most significant legal exposure.
It is also worth considering an assumption many teams make without examining it: that vector embeddings represent a safe, transformed representation of source data. Li and colleagues demonstrated in 2023 that sentence embeddings can be inverted to recover original source sentences. The vector store is a different format for storing content that, under the right conditions, can be reconstructed. It is not a privacy shield.
OWASP added LLM08:2025 covering vector and embedding weaknesses specifically because RAG's dominance created a vulnerability class the prior taxonomy didn't address. Academic research tracked the urgency in real time: publications on RAG security grew from fewer than five per year before 2024 to 18 in 2024 and 48 in 2025. When a research community accelerates that sharply, practitioners are usually already encountering problems that haven't found their theoretical grounding yet.
The practical implication is more procedural than technical. Access control in a RAG system must be enforced at retrieval time, scoped to what the requesting user is authorized to see. Enforcing it only at the output layer, after retrieval has already occurred, is too late.
What the Samsung, EchoLeak, and Slack AI incidents reveal about how these mechanisms play out in practice
Looking at these three incidents together is more instructive than treating any one of them as representative, because the mechanisms, the actors, and the failure modes are genuinely different in each case.
Samsung, March 2023
Samsung Semiconductor lifted its internal ChatGPT ban and within three weeks recorded three separate disclosure incidents. Engineers pasted proprietary source code into prompts. They submitted internal test sequences. One employee uploaded a transcript of a confidential meeting. No malicious actor was involved, and the model wasn't compromised. The engineers used the tool as it was designed to be used, and proprietary information went into OpenAI's systems.
What's notable about Samsung isn't the naivety of the employees. Samsung's own internal survey found that 65 percent of participants believed generative AI tools carried a security risk. The workforce had the intuition; the organization had no guardrails in place when the policy changed. A Fishbowl survey from May 2023 found that 68 percent of employees using ChatGPT at work did so without their employer's knowledge, which suggests shadow AI use was already widespread before most organizations had policies to govern it. Amazon and JPMorgan Chase imposed restrictions around the same period. Samsung was the most visible case of a systemic pattern that was happening everywhere.
EchoLeak, CVE-2025-32711, June 2025
The Microsoft 365 Copilot vulnerability assigned CVE-2025-32711 received a CVSS score of 9.3. A single crafted email triggered Copilot to access and exfiltrate internal files with no user interaction required. The attack chained several bypasses: it evaded Microsoft's cross-prompt injection attack classifier, circumvented link redaction, exploited auto-fetched images, and abused a Teams proxy that the platform's content security policy permitted. Potential exposure included chat logs, OneDrive files, SharePoint content, and Teams messages, essentially everything within Copilot's access scope at the enterprise level.
EchoLeak matters for a specific reason: it is the first documented case of prompt injection weaponized for concrete data exfiltration in a production enterprise AI system. The theoretical attack class became a real vulnerability in a widely deployed product. Microsoft patched it server-side and no in-the-wild exploitation was discovered, but the vulnerability existed in production before it was found. The technique generalizes to any LLM assistant with broad access to internal data.
Slack AI, August 2024
Researchers at PromptArmor embedded a prompt injection payload in a public Slack channel. When Slack AI conducted a search that included that channel in its context window, the injected instruction caused it to surface content from private channels. The attack vector was retrieval. This had nothing to do with training data; the sensitive content was in the context bundle, rather than the model's weights. Access to private channel content came through the RAG mechanism, via poisoning content the model would retrieve.
The Slack AI case is the clearest available illustration of context window leakage as a distinct mechanism. It required no access to the model's training process, no particular technical sophistication, no authorization. It required only the ability to post to a public channel in the same Slack workspace.
Samsung was accidental context disclosure, no attacker. EchoLeak was a sophisticated zero-click indirect injection in a production enterprise system. Slack AI was retrieval-based context leakage through a compromised document. The threat actor profile, the technical method, and the control failures differ in each case. No single defensive control covers all three, which is a significant operational implication for teams trying to prioritize.
How multi-agent architectures multiply the exposure surface beyond what single-agent analysis captures
The intuition many teams bring to multi-agent architecture is that decomposition reduces risk. If each agent has a narrower role and sees less data, each individual agent's exposure is smaller. That intuition is partially correct. The part that's correct is also where the real problem hides.
The AgentLeak benchmark, published in 2025, tested this at scale: hundreds of scenarios spanning healthcare, finance, legal, and corporate domains, run across five production LLMs including GPT-4o, Claude 3.5 Sonnet, and Llama 3.3 70B, generating thousands of validated execution traces. Multi-agent configurations did reduce final-output leakage, 27.2 percent versus 43.2 percent in single-agent mode. But total system exposure, accounting for internal channels including inter-agent messages, shared memory, and tool arguments, rose to 68.9 percent. Why exactly does this happen? Because narrowing each agent's visible output doesn't constrain what flows between agents in the channels connecting them.
The leak moves inward. Overall volume of exposure increases; it just becomes harder to see.
Monitoring the user-facing output layer of a multi-agent system, in practice, captures less than half of the system's actual exposure. Sensitive data passed between agents via tool calls or shared memory is frequently unmonitored and crosses trust boundaries without any audit trail. Each agent-to-agent communication is also a potential injection point: an attacker who can influence the content one agent processes can cascade that instruction to downstream agents, none of which have the context to recognize the input as adversarial.
The domains the benchmark examined are not abstract. Healthcare data, financial records, legal documents, government intelligence: these are all represented in real-world multi-agent deployments. The consequence of unmonitored internal channels in those domains is not a theoretical concern; it's a liability question.
Organizations deploying third-party agentic systems face an additional visibility problem. The external output is only part of the exposure picture. What flows between internal components, what data those components access, transform, and pass forward, is often opaque to the deploying organization. Vendor assessments scoped to output behavior miss the majority of the system's actual exposure surface, and most vendor assessments are scoped exactly that way.
The regulatory frameworks that now treat LLM-specific disclosure risks as compliance obligations
The core legal reality is straightforward: existing regulations were written for contexts other than LLMs and apply to them fully anyway. GDPR, HIPAA, and PCI-DSS each impose requirements that LLM deployments can violate through the mechanisms described above, and the fact that the mechanism is novel doesn't create an exemption.
GDPR creates several distinct pressure points. Purpose limitation and data minimization conflict with the practice of training on broad datasets containing personal data; the model's use of that data in outputs constitutes processing for purposes beyond what data subjects consented to. The right to erasure creates a technical problem with no clean solution. There is no established method for removing an individual's data from a model's weights after training. Differential privacy techniques can reduce the probability of memorization, but they don't eliminate it, and they are not equivalent to erasure. The legal obligation exists; the technical means to fully satisfy it does not.
HIPAA's minimum necessary standard maps directly onto the RAG access control problem. A RAG pipeline that retrieves health records the querying user was not authorized to see, and then summarizes that content in a model response, represents a disclosure that doesn't meet the minimum necessary standard, regardless of intent. Intent is irrelevant to the compliance analysis.
PCI-DSS's prohibition on storing sensitive authentication data in readable form has implications for any system prompt or RAG corpus that includes payment credentials, and for any fine-tuned model trained on data containing card numbers or authentication strings. That the storage medium is a model's weights rather than a database does not create an exemption.
What regulators have begun to recognize, and what the OWASP taxonomy reflects, is that LLM-specific disclosure mechanisms require LLM-specific risk assessments. Infrastructure security audits that don't evaluate the model's input/output surface, the retrieval pipeline's access controls, and the contents of system prompts are incomplete assessments under frameworks that require organizations to understand and govern how personal data is processed and disclosed. The perimeter has moved. The compliance obligation moved with it.
