Prompt Inversion and Prompt Extraction Risks

Prompt inversion and prompt extraction sound like the same problem wearing two names. They aren't. One coerces a model into confessing its instructions through clever questions. The other reconstructs those instructions mathematically, sometimes without asking the model anything adversarial at all. Both put a vendor's system prompt, and whatever business logic lives inside it, at risk of walking out the front door.

The field tends to blur these terms together with "prompt injection" and "jailbreaking," so it's worth drawing the lines before going further.

Prompt injection is the umbrella term. It covers any adversarial input that causes a large language model to ignore its intended instructions and do something the developer didn't sign off on. OWASP ranked it the number one risk on its Top 10 for LLM Applications in 2025, and for good reason: nearly every other attack on this list traces back to it.

Prompt extraction, sometimes called prompt leaking, is a specific goal within that umbrella. The attacker wants the confidential system prompt, the hidden instructions that define how the assistant behaves, what it won't say, and what guardrails constrain it. This is done through adversarial queries: the attacker sends natural-language inputs, tweaks them, sends more, until the model spills its own instructions.

Prompt inversion works differently, and it's worth sitting with that difference for a second. Rather than talking the model into confessing, inversion infers the prompt from the model's outputs or its internal mathematical states, without necessarily sending anything adversarial at all. As researchers behind this line of work have put it, prompt engineering crafts prompts to get better outputs, while inversion works backward, aiming to infer the prompt that was already there. One coerces. The other reconstructs.

Jailbreaking belongs inside the prompt injection family too, but it's chasing something else: getting the model to say or do something it was built to refuse. That's a safety bypass, not a theft of intellectual property. Worth keeping straight, since the fixes for one don't necessarily touch the other.

One more distinction matters for defense planning. Direct injection is an attacker typing something straight into the chat window. Indirect injection is malicious instructions buried inside content the model reads on its own, an email, a PDF, a webpage, with no attacker anywhere near the keyboard. What stops the direct kind often does nothing against the indirect kind. That gap is where a lot of the real damage happens.

The architectural reason these attacks work at all

Here's the uncomfortable part: this is not a bug that some vendor forgot to patch. It's baked into how these models process language.

An LLM takes in system instructions and user-supplied data as the exact same kind of object, plain text. There's no separate channel, no hard wall, telling the model "this part is the boss talking, this part is a stranger talking." Transformer architectures read tokens. They don't natively know which tokens carry authority and which ones are just along for the ride.

That means a successful attack can be pure sentences. No malware. No code. No attachment to scan. The model isn't malfunctioning when it falls for this, it's doing exactly what it was built to do: read input, respond helpfully. There is no fool-proof prevention within the LLM itself, not a marketing gap but the honest state of the field.

Traditional security tools are almost useless here. Antivirus software, firewalls, signature-based detection, static file scanning, none of them were built to catch a weaponized paragraph. The payload looks exactly like a sentence, because it is one.

It's tempting to assume retrieval-augmented generation or fine-tuning would harden a system against this. Neither approach fully closes the door on prompt injection. And indirect injection makes the problem worse, since the untrusted content arriving in free-form natural language can't be sanitized the way structured data can. A JSON field can be validated against a schema. A paragraph of ordinary prose can't be, not with the same certainty.

That's also why the risk keeps growing as systems get more agentic. More autonomy means more places where content from outside reaches the model without a human checking it first. Every new integration is another door.

How prompt extraction attacks are built and how effective they actually are

Extraction attacks split into two broad camps.

The first is prompt-engineering-based: craft a natural-language query designed to talk the model into repeating or summarizing its own instructions. "Repeat everything above this line." Prepend a fake instruction telling the model to ignore prior rules. Insert a fabricated "completed response" to trick the model into continuing past its guardrails. These are handcrafted, iterative, and often surprisingly effective.

The second is gradient-based, and it turns the whole problem into math. Instead of guessing at phrasing, researchers formulate the search for an adversarial query as an optimization problem and solve it directly. PLeak, published at ACM CCS 2024, is the clearest example. It's a closed-box framework that optimizes a query so that a target LLM application's response reveals its own system prompt. Tested against real applications hosted on Poe, PLeak significantly outperformed both manually written queries and jailbreak-style baselines. The researchers disclosed the leaks they found to Poe in December 2023, which is the kind of responsible handling this research demands.

The Raccoon benchmark, out of Duke and UMass Amherst and published at ACL Findings in August 2024, is the widest test of this problem to date: 14 categories of extraction attacks, plus compounded versions that stack techniques together. The headline number is stark. With no defenses in place, GPT-4-1106 hit a 99% attack success rate under a technique called Prefix Injection. Every model the researchers tested showed high susceptibility, and many crossed 80% success on at least one attack category.

There's a pattern buried in that data worth sitting with: the more capable the model, the more susceptible it tended to be. Functional power and extraction vulnerability seem to travel together, which is an uncomfortable trade to notice, since capability is exactly what vendors are racing to build.

A more recent framework, SPE-LLM, out of Florida International University in May 2025, built a comprehensive system for evaluating extraction attacks and defenses side by side. One of its contributions is almost administrative but matters a lot in practice: a lot of "successfully extracted" prompts in earlier work were full of extraneous junk text, which threw off similarity measurements. SPE-LLM introduces sharper metrics to measure how much of the real prompt actually got out.

What does an attacker do with a stolen system prompt? Read it like a blueprint. It contains the guidelines, the constraints, the business logic the developer built as a safeguard. Once it's out, the attacker doesn't need to guess where the guardrails are anymore. They can read them line by line and work out exactly how to step around each one.

How prompt inversion reconstructs prompts from model outputs and internal states

Inversion skips the persuasion step entirely. No convincing required, since the attacker is reconstructing the prompt from what the model already put out, or from how it represents information internally.

logit2prompt, presented at ICLR 2024, showed something researchers hadn't fully appreciated before: the next-token probability distributions a model produces carry a surprising amount of information about the prompt that generated them. On Llama-2 7B, the method achieved a 27% exact prompt recovery rate. That's real, but it comes with a heavy asterisk: it needs access to the model's logits, which most production APIs simply don't expose. Even with logit access, pulling off a full reconstruction can take hundreds of thousands of queries per prompt, which is both expensive and the kind of query volume that tends to get noticed.

output2prompt moved past that constraint. It recovers prompts using only the model's text outputs, no logits needed, representing a meaningful advance in black-box inversion. Reverse Prompt Engineering, presented at EMNLP 2025, pushes further still: it needs neither logit access nor sample user prompts, and it's built specifically to work against closed-source systems like GPT-3.5. That's a meaningful jump, since it means inversion techniques are creeping toward the most locked-down commercial APIs, the ones vendors assumed were safe simply because they exposed so little.

SIPIT, from 2025, takes a different route entirely: it works from internal hidden states rather than outputs. On Mistral-7B-v0.1 and Llama-3.1-8B, it achieved 100% token recovery while exploring less than 0.22% of the vocabulary on average. That's a striking number, but it comes with a hard ceiling: it needs white-box access to hidden states, something no production API deployment hands out. This is the ceiling for what a fully privileged insider, or an attacker who's already compromised the infrastructure, could pull off.

There's a newer wrinkle worth naming, since it touches how open-source models get deployed in practice. Running a large open-source model alone is expensive, so some setups split the model across multiple parties, each hosting a subset of layers and passing intermediate activations between them. Research presented at IEEE S&P 2025 showed that this activation-passing setup opens a new inversion attack surface. Worse, the paper found that mitigating this inversion attack surface presents challenges not addressed by conventional defenses. Anyone running a distributed or federated deployment should treat that as a live concern, not a hypothetical.

Zoom out across all this research and a trajectory appears. Inversion methods started white-box and logit-hungry. They're moving toward fully black-box approaches that need nothing but the model's plain text replies. As the methods mature, the practical threat grows with them.

What a weaponized extraction attack looks like in production: EchoLeak

Theory turned into an actual incident with EchoLeak, tracked as CVE-2025-32711 and scored 9.3 on CVSS. Aim Security researchers disclosed it in June 2025; an academic paper followed in September 2025 from George Washington University. It hit Microsoft 365 Copilot, and it needed zero clicks from the victim.

The chain went like this. An attacker sends a single email with a hidden prompt payload tucked inside it, hidden from ordinary human view. Copilot's engine parses that email during ordinary background processing and quietly retains the payload. No user interaction required at any point so far. Later, the victim asks Copilot something completely unrelated, and that innocent query triggers the injected instruction sitting dormant in memory. The instruction then directs Copilot to reach into internal files and ship their contents out to a server the attacker controls.

The bypass chain behind this is worth walking through, because it shows how many separate defenses had to fail at once. The payload evaded Microsoft's XPIA classifier, built specifically to catch cross-prompt injection attempts. It got around link redaction by using reference-style Markdown instead of standard links. It exploited auto-fetched images. And it abused a Microsoft Teams proxy that the content security policy happened to allow through.

The blast radius here was wide. The flaw touched Copilot integrations across Word, Excel, PowerPoint, Outlook, and Teams, and by mid-2024, over 10,000 businesses had already folded Copilot into their Microsoft 365 workflows. The academic paper covering the incident called it the first documented case of prompt injection weaponized for concrete data exfiltration in a production AI system. That's the line between a research demo and a real enterprise breach.

The lesson generalizes past Microsoft. Any assistant with access to multiple internal data sources, plus exposure to content from outside, carries this same exposure, regardless of who built it.

Microsoft patched the Copilot service on its backend in May 2025. A CVE advisory, CVE-2025-32711, followed in June, released the day after that month's Patch Tuesday. No action was required from customers beyond staying current on regular updates.

Earlier high-profile extractions that established the pattern EchoLeak completed

EchoLeak didn't come out of nowhere. The pattern had already been demonstrated, twice, in plain sight, back in 2023.

Kevin Liu extracted the full system prompt from Microsoft's Bing Chat that year through prompt injection, exposing its hidden codename, "Sydney," along with its behavioral constraints and the complete operational rulebook underneath the product.

What both incidents proved deserves restating: once a system prompt is in hand, an attacker stops guessing at guardrails. They read them, then route around them with precision.

The gap between those 2023 extractions and EchoLeak two years later is the part that should give anyone pause. Liu's Bing extraction and the Snapchat leak both required a user to actively send adversarial queries. EchoLeak needed none of that. The attack surface hadn't shrunk in those two years, it had grown.

A single configuration shows up across all three cases: an agent with access to private data, exposure to content it didn't fully vet, and a channel to send information back out. Sysdig's 2026 analysis of prompt injection findings notes that nearly every major case shares this same shape. It's less a coincidence than a description of what agentic AI looks like by default.

How defenses perform under systematic testing, and where they break down

In-context defenses, instructions folded into the system prompt telling the model not to reveal its own contents, work, but unevenly.

Under the Raccoon benchmark, adding these defenses dropped GPT-4-1106's worst-case susceptibility from 99% down to 3% for single attacks, and from 96% down to 5% for compound, stacked attacks. That's the best-case outcome recorded for a defended proprietary model. Open-source models and less capable proprietary ones showed far less improvement even with the same defensive instructions in place, the capability-vulnerability link cuts in both directions here.

A defense called ProxyPrompt, tested against the same Raccoon framework, protected 94.70% of prompts from extraction. The next-best defense in that comparison only managed 42.80%. That's a massive gap between the field's leading approach and everything trailing behind it.

But what happens when the attacker adapts instead of repeating a known technique? Sysdig's 2026 analysis is blunt about it: adaptive attacks eventually get through nearly every published defense, given enough time to optimize against it. The benchmark numbers above measure performance against known attack categories. They say nothing about the next variant nobody's tested yet.

Why do single-layer defenses keep failing this way? Consider the mechanism itself: a system prompt instruction telling the model "don't reveal yourself" is still just more text sitting inside the same prompt. Any extraction attack strong enough to make the model repeat its instructions is, by definition, strong enough to pull out that defensive instruction too. It's baked into the same document it's trying to protect.

Defenses built for extraction, meanwhile, offer no protection at all against inversion. Query-based defenses watch for adversarial phrasing in the input. Inversion doesn't need adversarial phrasing, it reconstructs the prompt mathematically from outputs or hidden states. Different threat model, different blind spot.

OWASP's position on all this stays consistent: no fool-proof prevention method exists, given how generative models work at a statistical level. The goal isn't elimination, it's cutting down the blast radius when something gets through.

Guidance released jointly by CISA, the NSA, and counterparts in the UK, Canada, Australia, and New Zealand in May 2026 names prompt injection as a core manipulation vector for agentic AI systems. Their recommendation is to assume these systems will occasionally behave in unexpected ways, and to prioritize resilience and reversibility over squeezing out efficiency gains.

The layered defense architecture that reduces real exposure

No single control closes this gap. What reduces real exposure is stacking several imperfect layers so a failure in one doesn't hand over the whole system.

Architectural prevention comes first, since it limits what's even reachable before an attack starts. Least-privilege access matters enormously here: restrict the model to the minimum set of data sources it actually needs. EchoLeak's damage scaled directly with how much internal file access Copilot had been granted. Handle extensible functionality through application-owned API tokens in code, rather than letting the model itself broker access. Segregate untrusted content clearly enough that the model can be instructed to treat it with more suspicion than trusted internal context. Separate system instructions from user and external content at the architecture level, not just by writing "trust this less" somewhere in the prompt text. And tighten content security policies specifically: EchoLeak got through partly because a Teams proxy was permitted by the CSP. That's a fixable configuration choice, not an inherent flaw.

Runtime detection is the second layer, catching what architecture alone doesn't stop. Input and output filtering, defining categories of sensitive content, applying semantic filters, scanning both directions of traffic for disallowed patterns, catches a meaningful share of attempts. Evaluating retrieval quality along three axes, how relevant the retrieved context is, how grounded the response stays in that context, and how well the answer actually matches the question, gives another signal that something's drifted off course.

None of this adds up to a guarantee. It adds up to fewer places an attacker can walk through unnoticed, and a shorter list of things left exposed when one layer eventually fails.