System Prompt Confidentiality and Leakage

Here is the thing nobody says plainly enough: a system prompt is just text. The model sees it as text, processes it as text, and can be made to reproduce it as text. That is the complete architecture. No encryption, no access control, no privileged execution context. The prompt sits in the same undifferentiated space as everything else the model ingests.

So what ends up in that layer matters enormously, and it varies a lot depending on who built the thing.

API keys and database credentials are probably the most immediately damaging category. A researcher reviewing nearly a thousand Flowise servers in 2024 found that 45% were vulnerable to an authentication bypass, CVE-2024-31621, that exposed plaintext passwords and API keys stored directly in system prompts. Live servers, not a controlled lab.

Authorization logic is subtler but more consequential at scale. When a system prompt defines who is allowed to do what, an attacker who reads it now understands the exact rules they need to circumvent. You have written out the guardrails for them, in plain language, in the same place they are going to look first.

Architecture details and named back-end service URLs are reconnaissance gifts of the most generous kind. An attacker who learns which database your agent queries, or which internal API it calls, has a targeted list of follow-on objectives handed to them. They did not have to map your infrastructure. You did it for them.

OWASP's LLM top ten, 2025 edition, LLM07, makes something important explicit: the disclosure of the system prompt itself is not the primary risk. The risk is what was delegated to it. Exposing a prompt that says "be helpful and concise" is categorically different from exposing one that contains an API key, maps your agent's tool permissions, and defines authorization rules that exist nowhere else.

There is another dimension that goes underappreciated even in security circles. An attacker does not necessarily need exact prompt text to extract value from what they find. Behavioral observation alone can surface guardrail structure. Ask a model something it is forbidden to answer, watch how it refuses, probe adjacent phrasings, and the shape of the constraints becomes apparent. The prompt does not have to leak verbatim to be useful intelligence.

For agentic systems, this expands further in ways that are alarming once you sit with them. A leaked agent prompt does not just reveal instructions; it reveals an operational capability map, which tools the agent can invoke, which APIs it connects to, what data stores are reachable, what actions it can trigger without further human authorization. Reading a chatbot's persona instructions is annoying. Reading that map is something else entirely.

How Attackers Extract System Prompts in Practice

The simplest attacks work against the most deployments, and I think that fact deserves more attention than it gets.

Instructing a model to "repeat everything above," or encoding tricks that ask the model to translate its instructions into Base64, reliably bypass naive output filters scanning for literal reproduction. These are not sophisticated techniques. They are the floor.

The more instructive data point is how much persistence matters relative to cleverness. Research from Salesforce AI Research, published at EMNLP 2024, studied ten closed- and open-source LLMs across four domains. A single extraction attempt succeeded at an average rate of 17.7%. Applying a multi-turn approach that exploited the model's sycophantic tendencies, its inclination to accommodate a persistent, escalating conversational partner, elevated that average to 86.2%. GPT-4 and Claude 1.3 reached 99% leakage rates under this approach.

Read that again: 99%. Determination matters more than cleverness, by a significant margin.

Indirect prompt injection is a different problem, and the more dangerous one operationally, because the attacker never directly interacts with the model at all. Malicious instructions embedded in web content, documents, email bodies, or issue titles can be processed by a model during normal operation. The model then executes those instructions as if they were legitimate input. The NCSC noted in late 2025 that this problem will not be fully resolved because it emerges from how large language models fundamentally process text. NIST characterized indirect prompt injection as generative AI's greatest security flaw. Neither organization phrased this as a theoretical concern.

One 2025 dataset documented over 461,000 prompt injection submissions, with success rates ranging from roughly 50% to 84% depending on technique. Automated extraction methods, gradient-based approaches and PLeak-style structured query sequences, allow systematic prompt recovery from production systems without any social engineering, requiring only repeated structured interaction and enough patience to analyze responses.

Real Deployments Where the Architecture Failed

In February 2023, simple user queries against Bing Chat extracted the internal system prompt, revealing codenames, operational rules, and Microsoft's underlying personality design for the assistant. At the time, a lot of people treated it as a curiosity, an amusing quirk of a new technology. In retrospect, it established the pattern that would repeat for years.

GitHub Copilot in 2024 demonstrated the same dynamic against a heavily resourced commercial product. Repeated extraction attempts surfaced proprietary instruction engineering and safety rules from a product that had invested significantly in confidentiality handling. Investment in confidentiality handling was not the same as confidentiality.

EchoLeak, CVE-2025-32711, is probably the clearest illustration of what downstream architectural failure actually looks like in production. Affecting Microsoft 365 Copilot, rated CVSS 9.3, zero user interaction required. An attacker sends a crafted email. The victim asks Copilot to summarize their inbox. Copilot silently exfiltrates sensitive documents to an external server. The user did nothing unusual. The vulnerability was not a missing patch in any conventional sense; it was a product that ingested external content, processed it through the model, and had insufficient separation between instruction context and content context.

CurXecute, CVE-2025-54135 and CVE-2025-54136, CVSS 9.8, involved malicious prompts embedded in a repository README for Cursor IDE. Opening the project caused the AI assistant to execute arbitrary commands. Remote code execution via indirect injection, through content that looked like documentation.

GitLab Duo passed issue titles directly into the model without sanitization. Crafted titles manipulated the assistant's responses and leaked internal project metadata. That sanitization gap was the entire attack surface.

A ChatGPT connector disclosed in August 2025 reads from Google Drive and SharePoint. Prompt injection in processed documents can be used to exfiltrate sensitive data from those connected applications. Unit 42 confirmed large-scale indirect prompt injection attacks against live commercial platforms in early 2026.

What connects all of these cases is not carelessness. Each product had invested in some form of prompt protection. The protection failed anyway, because the vulnerability was an architectural decision, not an absent patch.

What the Downstream Attack Chain Looks Like After a Prompt Is Extracted

Extraction is usually reconnaissance, not the objective itself.

The attacker who reads your system prompt now knows which guardrails exist and can design targeted bypass attempts. They know which back-end systems are named. They know what tools the agent can invoke, and they can map the privilege surface for lateral movement. A prompt that prohibits offensive content, external links, and code execution tells an attacker exactly three constraints to attempt to override, in the order they should probably try them. You have written the roadmap.

The temporal dimension is worth sitting with, because it changes how you think about risk accumulation. A single direct extraction attempt succeeds approximately 17.7% of the time without safeguards. By the two-hundredth attempt, the breach rate reaches a very high level. An agent deployed in a high-volume workflow accumulates interaction opportunities that translate directly into elevated breach probability over time. The attack surface grows with use, not only with sophistication.

The International AI Safety Report 2026 found that sophisticated attackers bypass the best-defended models roughly half the time with just ten attempts. Ten. That is the empirical context for any security control relying on the model's resistance as its primary mechanism.

For agents with broad tool access, a leaked prompt is functionally a privilege map: which APIs are accessible, what data can be reached, what actions can be initiated without further authorization checks. The instructions reveal not just what the model was told, but what it can do. Those are different categories of exposure, and treating them as equivalent is a mistake that shows up repeatedly in how organizations assess their AI risk.

Why the Standard Response, Hiding the Prompt, Doesn't Address the Problem

The intuitive move is to add an instruction telling the model not to reveal its system prompt. Some developers rely on provider-level confidentiality handling and call it a day. Both responses miss the structural problem by a wide margin.

OWASP is direct on this: training a model not to reveal its system prompt is not a guarantee of adherence under adversarial pressure. An 86.2% average multi-turn success rate is not an edge case you can bracket off. It is the expected outcome for a determined attacker at scale.

A June 2026 study measuring prompt leakage across hundreds of applications on six major commercial platforms found that prompt engineering defenses preserve usability but offer limited leakage resistance. Output-based detection achieves better protection, but at meaningful cost to usability. There is no prompt engineering approach that resolves both simultaneously, and anyone telling you otherwise is selling something.

Hiding instructions does not remove credentials from the prompt. It does not move authorization logic into deterministic systems. It does not prevent an attacker who successfully extracts the prompt from exploiting everything stored in it. The confidentiality instruction is a soft barrier on a layer that was not hardened to begin with.

Provider-level policies are also not a substitute for design choices, and this is a point that gets glossed over in vendor conversations. OpenAI retains Enterprise chat logs by default for a period of time for abuse monitoring. Prompts exist in logs regardless of what in-conversation confidentiality instructions say. Shadow AI, employees using personal accounts for company-related work, introduces consumer data handling terms where enterprise agreements were assumed. Neither scenario is addressed by a prompt that says "do not repeat these instructions."

What Sound Design Looks Like When System Prompts Cannot Be the Security Layer

The core principle, drawn from OWASP's guidance, is to avoid using system prompts to control security-relevant behavior wherever possible, and to rely on systems outside the model to enforce it. That sounds obvious stated plainly. It is apparently not obvious enough, given the frequency with which it gets violated.

Authorization logic belongs in deterministic back-end code, not in a role definition embedded in a prompt. If your application needs to enforce that users with a particular role cannot access certain data, that enforcement should happen before the model is invoked, independently of what the model was told. These are not the same control, even if they produce the same outcome under benign conditions.

Credentials should not enter the model's context at all. Secrets managers exist precisely for this. Credentials resolved at the infrastructure layer, before they reach the model, cannot be reproduced by the model because the model never saw them. That is a clean architectural boundary, and it is achievable without significant complexity.

OWASP's guidance on output inspection is worth taking seriously: an independent system that inspects the model's output for compliance with expectations is preferable to system prompt instructions. The inspection is the control. The model's compliance is not. Conflating those two things is the root of most of the failures described above.

Input sanitization matters at every boundary where the model ingests external content. Documents, web pages, email bodies, issue titles, anything sourced from outside the application's direct control should be treated as potentially adversarial before it reaches the model context. GitLab Duo's issue title problem was a sanitization gap. The fix was to sanitize, not to instruct the model to be cautious.

For agents specifically, least privilege applies to tool permissions just as it applies everywhere else in security. The prompt should not enumerate every capability the system possesses. Tool access should be scoped to what a specific task requires, granted at invocation, and revoked after.

Treat the system prompt itself as an auditable artifact. If it contains something you would not want an adversary to read, it should not be in the prompt. That is not a security control; it is a design review criterion. And continuous monitoring matters in a way that periodic review cannot replicate: the attack surface changes with model updates, connector additions, and expanding agent capabilities. A deployment that passed a review three months ago looks different today.

How Vendor AI Deployments Extend This Problem Beyond Internal Development Teams

Most enterprise AI exposure does not come from models the organization built. It comes from vendor-supplied assistants, plugins, connectors, and agents integrated into existing workflows: productivity tools with AI embedded, customer service platforms, code assistants, document processing pipelines. The security team often has no visibility into what those deployments contain or how they behave.

Each vendor integration carries a system prompt the enterprise did not write, cannot audit in most cases, and may not know exists. EchoLeak and the ChatGPT connector incident illustrate how a third-party AI feature becomes the actual attack vector. In both cases, the enterprise had no direct visibility into what the underlying prompt contained, what data the agent was authorized to access, or what it would do when it encountered crafted content in its input stream.

Traditional third-party risk management frameworks assess vendors on data handling policies, compliance certifications, and contractual commitments. They are not designed to evaluate whether a vendor's AI system prompt embeds credentials, whether authorization logic is enforced in deterministic systems or delegated to model instructions, or whether the product is vulnerable to indirect injection through content it processes. These questions do not appear on a standard questionnaire because the frameworks predate the architectural patterns that create this exposure.

What meaningful assessment of vendor AI actually requires is understanding what a vendor's AI components can access and act on, whether authorization logic is deterministic or model-delegated, and how the risk profile shifts as the vendor updates its model or expands its agent capabilities. That last point is hard to address with conventional third-party risk management: the assessment completed three months ago does not reflect what the product does today, and vendors update continuously.

Platforms combining vendor risk assessment with continuous monitoring of live AI deployments can surface prompt injection exposure, agentic capability maps, and connector risk as they exist at a given moment, not just at the time of a scheduled review. Promptarmor is designed to do exactly this, continuously monitoring vendor AI assets to surface what is embedded in system prompts across an organization's ecosystem, giving security and TPRM teams visibility into which deployments are treating prompts as a security boundary before an attacker discovers the same thing.

The threat is not primarily that someone reads your instructions. The threat is that the instructions contained something exploitable, that no deterministic control exists outside of them, and that the model can be induced to reveal them through techniques that are documented, automated, and increasingly routine. Hiding the instructions more carefully is a response to the wrong problem. The exposure is architectural, and that is where it has to be addressed.