Architectural Controls That Make System Prompt Contents Irrecoverable to End Users

System prompt confidentiality cannot be guaranteed by asking an LLM to keep its instructions secret. The strongest protection keeps secrets out of model context, then adds least-privilege access, independent output inspection, and connector monitoring outside the model. Detecting extraction attempts also requires watching repeated probing queries, tool activity, and direct requests for the prompt. These controls can make extraction practically infeasible, but prompting alone cannot make it impossible.

Why system prompts cannot be made secret by the model that holds them

Microsoft’s “Sydney” chatbot incident showed how little an attacker might need to begin. Its system prompt was extracted not through a server exploit, but simply by asking in the right way. The model held instructions that users were not meant to see, yet could reproduce them in conversation. That is the foundational problem with prompt secrecy. The component expected to keep the information confidential also processes requests to disclose it.

LLMs process system instructions and user input in the same stream, with no built-in trust boundary that reliably keeps privileged instructions separate from adversarial content. A direction not to disclose information is therefore not equivalent to an access control enforced outside the model. Refusal can make disclosure less likely, but it cannot rule it out. Keeping any API key, credential, internal URL, or confidential policy logic in that context means trusting the model to keep a secret it cannot reliably keep.

Impossibility would mean no request could recover protected content. Practical protection instead limits what the model knows, what its tools can reach, and what information can leave through its responses. The architectural goal is to reduce exposure without treating successful refusals as proof of confidentiality.

What attackers are actually doing to extract prompt contents

Direct attacks try to change how the model understands the request. Role-play, jailbreaks, and encoding tricks can present disclosure as a different task rather than an explicit demand for hidden instructions. The wording is not the only variable. An attacker may change the requested operation itself, asking the model to format protected content.

Indirect injection removes the need to address the assistant directly. Aim Labs’ 2025 EchoLeak finding in Microsoft 365 Copilot, CVE-2025-32711, showed that sensitive data could leave the tenant with zero user clicks, the first documented zero-click exfiltration from a production enterprise AI assistant. Reprompt, CVE-2026-24307, demonstrated single-click data exfiltration from Microsoft Copilot Personal through URL parameter injection, without user-entered prompts. Emails, documents, web pages, and tool outputs supplied as material for the assistant to process can be treated as instructions.

ZombieAgent exploited ChatGPT’s connector integrations and long-term memory to achieve zero-click indirect prompt injection that survived session boundaries. Untrusted project content gives attackers more ways to reach coding agents. Claude Code, CVE-2025-55284, was exploited through DNS-based exfiltration following an indirect injection payload in its context window. GitHub Copilot, CVE-2025-53773, suffered remote code execution through an indirect injection in a repository file, running with the developer’s user-level permissions.

Encoding attacks most directly challenge reliance on refusal. Keysight Technologies’ research at IEEE Cyber-AI 2026, by Sahu, Samanta, and Soosahabi, tested seven common models against 46 verified system instructions. Attack success rates were at or above 0.7 for structured serialization, even where models refused direct extraction. The same content remained recoverable under a different task. This is why an apparently well-configured refusal policy still needs controls outside the model.

Why the attack surface has expanded beyond the system prompt text itself

OWASP’s 2026 LLM Top 10, published August 3, 2026, replaced System Prompt Leakage with Hidden Context Exposure, ranked LLM08 [12][1][2][4][5][11][3][6]. Hidden context includes retrieved documents, agent memory, user information, application state, and tool responses. Each can hold information that users should not see. Protecting one instruction block leaves the rest of that information surface unresolved.

Agentic deployments also put more operational detail into prompts. Tool descriptions, workflow definitions, access APIs, and sometimes secret validation tokens designed to detect injection can all appear there. Disclosure can therefore reveal more than a chatbot’s behavioral instructions. It can expose how an application works and what resources its agents use. That can make a leak more harmful even when the prompt has no conventional password.

But what if the attacker never recovers the prompt? SPORE, published April 2026, illustrates why that is not enough. The method requires no training and extracts private information directly from an LLM agent’s memory during inference. A memory store, vector index, or chained tool response can expose information as damagingly as the original instructions. The unit of protection must therefore be the whole in-context information surface.

What MCP servers and connectors add to the exposure problem

MCP servers introduce third-party content through a channel separate from the model but connected directly to its context. Their risks include poisoned tool descriptions, unauthenticated calls, malicious or vulnerable packages, indirect injection through tool results, and over-broad permissions. These are not all prompt-writing problems. Some concern the software and access controls that supply the model with information and capabilities. In-application refusals cannot substitute for governing those components.

Simon Willison documented in April 2025 how tool descriptions visible to the model but absent from user interfaces could contain hidden adversarial instructions. Invariant Labs demonstrated cross-server tool shadowing, in which one malicious MCP server weaponized adjacent trusted servers through manipulated descriptions. Separately, mcp-remote suffered command injection in a widely used MCP proxy component, CVE-2025-6514, with CVSS 9.6. Connector exposure includes both injected instructions and conventional software vulnerabilities.

Approval creates a further question about time. A server can behave normally during assessment, then change its tool descriptions or behavior after trust has been granted. CyberArk’s 2025 research established that any server-controlled field, including tool names, descriptions, arguments, results, and error strings, can be an injection channel. A standard point-in-time approval does not catch that later “rug pull.” The control must extend beyond deciding whether the server was safe when it first connected.

The architectural controls that actually reduce extraction risk, ordered by effectiveness

The first control is to remove secrets entirely. API keys, credentials, connection strings, tokens, and internal URLs belong in a secrets manager referenced at runtime. OWASP guidance puts more weight on making disclosure harmless than on making the prompt harder to recover. This is the only universally effective control against extracting those secrets from the prompt, because the prompt no longer contains them. It does not, by itself, protect sensitive information retrieved later.

Least-privilege architecture replaces a single “god-mode” agent with multiple agents, each limited to the access its task requires. Where tasks need different access levels, separate configurations constrain what each agent can reach or expose. Narrow permissions must be enforced. OWASP moved Excessive Agency from sixth to third place in its 2026 ranking, and its incident-weighted methodology gives that shift weight beyond expert opinion alone.

The third control inspects what leaves the model. An independent runtime output check can flag responses that repeat system prompt content, so the model giving the response does not have to police itself. Watching for repeated extraction attempts adds a second signal, because probing queries can show an attacker learning how the defenses work before making a targeted attempt.

The Agent Control Standard, donated to the OWASP GenAI Security Project and announced September 1–2, 2026, describes a complementary runtime governance approach. It defines an Agent Control System with guardian-agent enforcement points, observability through OpenTelemetry and OCSF, and an Agent Bill of Materials exposing tools, models, and accessible data. Its maturity needs careful qualification, however. The initial version defines the core terms, while v1 aims to add instrumentation and reference guardian-agent samples. Version v2 plans full AgBOM component-format support through CycloneDX, SWID, and SPDX, and v3 plans expanded protocol support across agent-to-agent communication protocols and MCP. Its inspectable, traceable, and instrumentable properties provide useful audit criteria.

The fourth control reduces the reconnaissance that remains. Keysight’s research showed that one-shot instruction reshaping with a reasoning model could make extraction much less likely to succeed through small changes to wording and structure, without model retraining. Unlike the independent controls, this hardening still works inside the model’s instruction processing. Its benefit is therefore probabilistic. It belongs beneath stronger architectural safeguards.

The fifth control governs the connector channel itself. An inventory of MCP servers, an allowlist of approved servers and tools, least-privilege credentials, and runtime inspection of tool calls and results address exposure that output filtering alone does not cover. Inspection belongs at the network or runtime layer. The model cannot reliably identify malicious instructions inside the tool content it is processing. Watching approved servers for later changes in behavior keeps checks going while the servers are in use.

What governance frameworks do and do not cover for prompt exposure risk

NIST AI RMF provides governance and risk management structure, but it does not specify or require runtime controls. An organization can satisfy its requirements while agents make ungoverned tool calls without an audit trail. Documentation and enforcement serve different purposes here. An AI inventory and risk register establish what an organization recognizes and manages. They do not inspect what an agent does with its granted tools.

MITRE ATLAS addresses adversarial threats and attack surfaces, while NIST AI RMF, NIST AI 600-1, and NIST CSF 2.0 provide governance and control structures. The Treasury Department Financial Services AI Risk Management Framework, released February 19, 2026, builds on NIST’s structure in partnership with the Cyber Risk Institute. It has 230 control objectives for the AI lifecycle, with a section devoted to third-party risk. That specificity helps organize responsibility, but runtime protection still needs implementation.

DORA Article 28 places full responsibility for managing ICT third-party relationship risks on financial entities. Generative AI platforms, inference services, and AI-enabled SaaS tools fit its definition of ICT third-party service providers. The European Supervisory Authorities published the first official list of 19 Critical ICT Third-Party Service Providers under Article 31(9) on November 18, 2025. The list includes AWS, Microsoft, Google Cloud, Deutsche Telekom, Oracle, and SAP. Outsourcing the service does not outsource the responsibility. The practical response is to connect governance obligations to the access restrictions, inspection, and monitoring described above.

Why vendor AI assets require continuous monitoring, not point-in-time assessment

Vendor AI exposure changes after procurement and deployment. Model versions, policies, default settings, and connector behavior can shift without notice, while retention and training defaults differ by plan tier and can change unilaterally. An initial assessment cannot establish permanent safety under those conditions. The same problem applies to connector approval, where a later update can alter what the model receives. Continuous monitoring must cover vendor AI assets as well as the application’s own controls to track these changes.

The 2026 State of Third-Party Risk Management Survey found that AI tied with cybersecurity as organizations’ top third-party risk concern for the first time. A majority of institutions were only partially aware of which vendors used AI, and no surveyed organization felt extremely confident managing AI-related risks. Teams cannot reliably connect runtime protection to vendor risk management when they do not know where AI enters their ecosystem.

Dedicated AI risk intelligence platforms address this gap by combining vendor risk assessment with real-time AI threat detection across the vendor ecosystem. Their relevant criteria are continuing visibility into vendor changes and emerging threats. Queries, outputs, and tools still each need runtime inspection. A platform’s role is to connect that changing exposure to compliance and business continuity risk.

Home365 illustrates the accountability at the end of that chain. The company deployed AI rather than building it, yet was held accountable. For TPRM, information security, privacy, and Legal teams, vendor ownership therefore does not make ongoing monitoring optional. The responsibility follows the deployment, including what changes after approval.