LLM Threat Modeling for Enterprise Applications

Start with the most fundamental thing: an LLM cannot reliably distinguish between instructions it's supposed to follow and content it's supposed to process. That single property changes everything about how you reason through risk.

In a traditional application, a trust boundary is enforced by code. An unauthenticated user hits an endpoint; the authentication middleware either passes or rejects them. The logic is deterministic. You can model it, test it, verify it. With an LLM, the "enforcement" happens inside a probabilistic model trained to be responsive to natural language. You can write a system prompt that says "do not reveal user data." You can also write a cleverly phrased user message that causes the model to reveal user data anyway, and no line of code was exploited in the process.

I've watched security engineers hit that realization mid-review and not know where to put it. It doesn't fit the mental model they've spent years building. That disorientation is actually informative, because it points to what's structurally new here.

Trust boundaries in LLM systems require explicit mapping before any threat modeling work begins. The key boundaries: user input flowing into the model's context window; retrieved external content from RAG pipelines, web searches, uploaded documents, or knowledge bases; model output flowing into downstream execution environments like code runners, SQL interfaces, and external APIs; agent-to-agent communication in multi-agent pipelines; training and fine-tuning data flowing into model weights; third-party plugins, connectors, and Model Context Protocol servers flowing into the model's tool use.

Every one of these is a potential injection or exfiltration point. But what if attackers don't need internal access to the system at all? They need access to something the model reads and trusts — and that's a different threat premise than most enterprise security programs are built around.

Agentic deployments make this considerably more complicated. Each tool integration an agent carries, every API it can call, every database it can query, represents a new trust boundary. The attack surface scales directly with capability, which means the more useful you make the system, the larger the exposure you're managing.

Your threat model has to begin with a data-flow diagram that traces every trust boundary, not just the network perimeter. If you haven't mapped where information enters the model's context and where the model's output goes, you haven't started threat modeling yet.

The Primary Reference Taxonomy: What the OWASP Top 10 for LLM Applications Covers and Why It Anchors Enterprise Threat Modeling

OWASP's Top 10 for LLM Applications is maintained by over 600 contributing security experts and was updated in 2025 to reflect actual production deployment patterns, including RAG pipelines and agentic systems. There are other references worth knowing. This one earns its anchor position because it maps to failure modes that have actually occurred in production, not theoretical attack classes constructed in a lab.

The ten categories, and what they cover in practice:

LLM01: Prompt Injection. Attackers override model instructions through direct user input or indirect content: documents, emails, RAG-retrieved chunks, tool outputs. It holds the top position because it's the most fundamental and hardest to fully prevent. We'll spend more time on it later.

LLM02: Sensitive Information Disclosure. The model memorizes and reproduces training data fragments, including PII, proprietary content, and confidential documents. This category jumped from sixth to second in the 2025 edition. That move reflects frequency in production incidents, not a theoretical reclassification.

LLM03: Supply Chain. Compromise can enter through base models, fine-tuned models, datasets, libraries, orchestration frameworks, vector databases, plugins, or cloud infrastructure. The attack surface here is broader and less visible than traditional software supply chain risk. Most organizations have significantly less visibility into model provenance than they assume.

LLM04: Data and Model Poisoning. An attacker manipulates training data, fine-tuning sets, feedback loops, knowledge bases, or RAG document stores to influence future model behavior. The manipulation can be subtle and may not surface for a long time.

LLM05: Improper Output Handling. Downstream systems execute model-generated SQL, HTML, shell commands, or API calls without validation. Code generation assistants are a particularly acute version of this: a model can produce a backdoored function that a developer reviews, approves, and ships without detecting the vulnerability. The model didn't make an obvious error. It made a persuasive one.

LLM06: Excessive Agency. The model is granted autonomy beyond what the task actually requires. Its scope, permissions, or action space exceeds those requirements. This category received significant expansion in 2025, which tracks with how broadly agentic deployments have scaled.

LLM07: System Prompt Leakage. New in the 2025 edition. Attackers extract internal rules, filtering logic, and permission structures from the system prompt. OWASP's guidance here is unambiguous: do not treat the system prompt as a security control.

LLM08: Vector and Embedding Weaknesses. Poisoned vector stores, insufficient access controls across tenant boundaries, and manipulated embedding models that degrade RAG trustworthiness. This is an authorization problem that gets classified as an AI problem, which means it sometimes falls between teams.

LLM09: Misinformation. Renamed from "Overreliance" in 2025. The scope expanded from users over-trusting model outputs to the model actively generating and propagating false information. The rename matters because it shifts the framing from user behavior to system behavior.

LLM10: Unbounded Consumption. Uncontrolled resource usage enabling denial-of-service, financial exploitation via pay-per-use APIs, or unauthorized model replication.

The limitation of this taxonomy is equally important to name: knowing these categories exist doesn't tell you how to apply them to your specific architecture. A taxonomy is not a methodology. You still need a process.

Using MITRE ATLAS Alongside OWASP to Map Adversary Techniques and Tactics

OWASP names the vulnerability class. MITRE ATLAS tells you what an adversary actually does to exploit it. Those are different kinds of knowledge, and you need both.

ATLAS is an adversarial machine learning knowledge base modeled on the ATT&CK framework, documenting tactics, techniques, and procedures specifically targeting AI and ML systems. As of v5.1.0, released in November 2025, it contains 16 tactics, 84 techniques, 56 sub-techniques, 32 mitigations, and 42 real-world case studies. The case study library is where I'd point anyone getting started: concrete examples of how attacks have unfolded in practice are more useful for detection engineering than abstract technique descriptions, full stop.

The November 2025 update added material directly relevant to enterprise agentic deployments, including a new Command and Control tactic and 14 new techniques covering memory manipulation, thread injection, RAG credential harvesting, and tool invocation attacks. The spring 2025 release added RAG Poisoning, False RAG Entry Injection, LLM Prompt Crafting, Impersonation, and AI Supply Chain Compromise. The cadence of updates matters here. This isn't a framework that was written once and left standing. It's tracking active adversary behavior.

ML supply-chain compromises rank among the highest-likelihood initial access vectors in ATLAS case studies for 2025. Why exactly does this happen? An organization deploying a fine-tuned model inherits risk from every node in that model's provenance, and most organizations have limited visibility into that provenance. They often can't tell you where the base model came from, let alone the training data.

Where ATLAS adds the most operational value is in giving security teams ATT&CK-style detection and hunting hooks. If you're running a threat hunting program, ATLAS lets you map adversary behavior across the kill chain: initial access, execution, exfiltration. That's the difference between a framework you reference occasionally and one you actually build detection logic against.

The key limitation to name: ATLAS is a potent threat intelligence tool, but it does not provide risk prioritization or impact-weighted scoring. It tells you what adversaries do; it doesn't tell you which of those things matters most for your specific deployment. Pair it with a scoring approach, or the enumeration becomes its own problem.

Adapting STRIDE for LLM Systems and Where It Falls Short Without Extensions

STRIDE isn't wrong for LLM systems. Some mappings are clean. Tampering maps to prompt injection, where an attacker modifies model inputs or retrieved context. Elevation of Privilege maps to Excessive Agency. Denial of Service maps to Unbounded Consumption. Information Disclosure maps to Sensitive Information Disclosure and System Prompt Leakage. Teams that already know STRIDE can orient themselves quickly using those mappings, and that familiarity has real value.

The problem is what STRIDE leaves unaddressed.

Context poisoning, where an adversary corrupts an agent's memory so it makes progressively worse decisions across future sessions, doesn't fit cleanly into any STRIDE bucket. Reasoning subversion, manipulating the model's chain-of-thought or tool-selection logic, isn't addressed. Unsafe tool use, tricking an agent into misusing legitimate permissions to delete files or exfiltrate data through indirect injection, falls into a gap between several STRIDE categories without landing squarely in any of them. You can feel the framework straining when you try to apply it to these scenarios. Things don't resolve the way they're supposed to.

A research extension called ASTRIDE, published in December 2025, addresses this directly. It extends STRIDE with three new categories designed specifically for agentic systems: Prompt Injection, Context Poisoning, and Unsafe Tool Use. It also automates threat report generation from architecture diagrams, which reduces manual overhead in the modeling process.

Separately, a STRIDE-AI synthesis framework integrating MITRE ATLAS, the NIST AI RMF, and the OWASP LLM Top 10 into a six-phase assessment lifecycle showed meaningful results in a validation case study: structured mitigation reduced attack success rate from 80% to 15%. That's a substantial improvement, and it illustrates what a layered approach can accomplish versus applying any single framework in isolation.

For scoring, DREAD, which covers Damage, Reproducibility, Exploitability, Affected Users, and Discoverability, remains widely used despite Microsoft deprecating it internally due to inconsistent scoring. The inconsistency problem is real but solvable: apply it with defined scoring rubrics and calibrate those rubrics across your team before you start. The framework is useful; unanchored scoring is the failure mode, not the framework itself.

Use STRIDE as your starting structure, extend it with ASTRIDE categories or an explicit OWASP overlay, and treat neither as complete on its own.

Prompt Injection as the Threat That Cuts Across Every Layer of the Stack

Prompt injection holds the top OWASP position for a reason that becomes clearer the longer you work with these systems. It's not just another vulnerability class. It's the consequence of the fundamental architectural property described earlier: the model cannot natively distinguish attacker instructions from legitimate ones. Every mitigation you build exists inside a system with that property at its core. There's no patching your way out of it structurally. You manage it; you don't eliminate it.

There are two vectors to model separately, because they have different attack surfaces and different mitigations.

Direct prompt injection: a user types malicious instructions into the chat interface. This is the version most people think of first, and it's the easier one to address, relatively speaking, because you can apply input filtering, context isolation, and output validation at the user-facing boundary.

Indirect prompt injection is harder, and in my experience, it's the one that gets underestimated. The model reads attacker-controlled content from documents, emails, web pages, code repositories, tickets, knowledge bases, or RAG-retrieved chunks. The model has no way to verify provenance. It processes the content because that's what it's designed to do. An attacker who can influence what the model reads can influence what the model does, without any direct access to the system. But what if the most dangerous injection surface isn't the user interface at all — but the document store your team considers internal and safe?

In 2025, prompt injection matured well beyond simple jailbreaks. Hybrid attacks emerged that blend language manipulation with classic exploitation techniques, using prompt injection as the initial vector and reaching data exfiltration or remote code execution as the terminal objective. Adversaries discovered that the indirect vector is often completely unmonitored, and they adjusted accordingly.

Agent-to-agent communication introduces another indirect vector that deserves its own treatment. Malicious instructions can propagate across a multi-agent pipeline through inter-agent messages. An instruction injected into the first agent's context can influence the second agent's behavior, and on through the chain. In a complex agentic deployment, this creates a lateral movement path that's difficult to detect from any single monitoring point, because no individual hop looks obviously malicious.

A real-world pattern that illustrates the stakes: a "second-order" prompt injection scenario where a low-privilege agent was fed malicious content that escalated into broader unauthorized access. The escalation happened not because permissions were misconfigured in the traditional sense, but because the agent's action space was wide enough that a manipulated decision led to high-impact consequences. The injection was small. The blast radius wasn't.

Check Point's 2026 Cybersecurity Report found that 40% of MCP servers were vulnerable. That matters because MCP servers are a direct indirect injection surface when agents use them to retrieve external content.

The threat modeling requirement that follows: every content source the model reads must be treated as an untrusted input in your data-flow diagram. Not "mostly trusted" or "internal so probably fine." Untrusted. For teams accustomed to treating internal document stores as safe by default, that's a meaningful operational change, and it's one that generates pushback. The pushback is understandable. It doesn't change the threat model.

Data Exfiltration Paths That Enterprise Threat Models Routinely Miss

A 2025 LayerX industry report found that 77% of enterprise employees who use AI have pasted company data into a chatbot query, and 22% of those instances included confidential personal or financial data. Most of that activity is invisible to conventional DLP tools, because the exfiltration doesn't look like a file transfer or an unusual outbound connection. It looks like a user doing their job.

That behavioral risk is difficult to address through generic DLP. The data leaves through the model's input channel, and most DLP monitoring is oriented toward output channels. The mismatch isn't a configuration problem. It's an architectural one.

There are three more specific exfiltration paths worth modeling explicitly, because each has a different attack pattern and a different mitigation surface.

Training and fine-tuning memorization, LLM02 in OWASP's taxonomy: the model reproduces PII or proprietary content from training data in response to targeted queries. This is a property of the model itself, not the application layer. It can't be patched away after the fact. It requires controls during the training pipeline: differential privacy, membership inference testing, data governance on what goes into fine-tuning sets. By the time it's in production, the window for the most effective mitigations has already closed.

RAG retrieval leakage, LLM08: insufficient access controls on vector stores expose sensitive documents across tenant or role boundaries. If your RAG pipeline retrieves based on semantic similarity without enforcing document-level permissions, a user query can surface content they should have no access to. Poisoned vector stores can be used to retrieve cross-tenant data deliberately. This is fundamentally an authorization problem, but it gets classified as an AI problem, which means it sometimes doesn't get owned by anyone clearly.

Output channel exploitation, combining LLM05 and LLM01: prompt injection causes the model to embed sensitive data in its output, in rendered HTML, in API calls, in generated code, exfiltrating it to attacker-controlled endpoints. The injection happens at one layer and the exfiltration happens at a different layer, which makes the end-to-end attack hard to detect from any single monitoring point.

Five major LLM-related data breaches occurred globally in just the first two months of 2025, with weak access controls around LLM inputs and outputs as the recurring pattern. That's not a coincidence; it's a structural vulnerability in how these systems are typically deployed.

The threat modeling implication: trace exfiltration paths at the output boundary. What systems render or execute model output? Does any of that output leave the enterprise perimeter? Vendor and third-party deployments add another layer, because enterprise data flowing into a vendor-hosted model is entirely outside the organization's direct visibility. Traditional third-party risk management processes generally don't assess at the model level. Most vendor questionnaires weren't written with this threat surface in mind.

Agentic AI as a Force Multiplier for Every Threat Category Already Modeled

Agents don't just generate text. They browse the web, execute code, query databases, call external APIs, and trigger real-world actions. The practical consequence: the blast radius of any single threat expands dramatically when the model has real tools at its disposal.

OWASP recognized this directly, releasing a separate Top 10 for agentic AI systems in late 2025 and describing 2025 as "the year of LLM agents." LLM06, Excessive Agency, received significant expansion in the updated edition. These aren't cosmetic updates.

Consider how threat severity shifts under agentic deployment. Prompt injection can now trigger lateral movement, data deletion, and API abuse, not just manipulated text output. Supply chain risk scales with the number of tool integrations: each plugin, connector, and MCP server is an additional supply-chain node. Excessive agency becomes harder to audit when an agent's action space is defined at runtime rather than at design time. Memory manipulation and context poisoning, for which MITRE ATLAS added specific techniques in November 2025, turn an agent's persistent memory into an attack surface that persists across sessions and across users.

That last point is worth slowing down on. In a stateless LLM application, each conversation starts fresh. In an agentic system with persistent memory, an adversary who successfully corrupts the agent's memory in one session will influence its behavior in future sessions involving entirely different users or tasks. The attack doesn't need to succeed completely in a single attempt. It can accumulate. That raises an important question: is your monitoring program designed to detect threats that accumulate gradually across sessions, or only threats that succeed in a single attempt? That's a different kind of threat persistence than most enterprise security programs are designed to detect.

The threat modeling requirement for agentic systems: map the full action space of each agent, every tool it can invoke, every permission it holds. Apply least-privilege principles to that action space specifically, not just to the network layer. The agent itself has a permission surface, and that surface is frequently over-provisioned relative to what the actual task requires. In my experience, the over-provisioning isn't deliberate; it happens because it's easier to give an agent broad permissions during development and harder to remember to scope them down before deployment.

A Six-Phase Process for Systematically Working Through an LLM Threat Model

Phase 1: Scope and Architecture Mapping

Document the full deployment stack: model provider, fine-tuning pipeline, RAG components, vector store, plugins, agent tools, connectors, output consumers. Every node. Then produce a data-flow diagram that marks every trust boundary identified earlier in this piece.

Avoid abbreviating this step. The data-flow diagram is the foundation everything else builds on. If your diagram doesn't show where data enters the model's context and where the model's output goes, you'll miss threats that only become visible when you trace the full path.

Phase 2: Threat Enumeration Using OWASP and ATLAS

Apply the OWASP LLM Top 10 categories to each trust boundary in your diagram. For each OWASP category that applies, cross-reference ATLAS to identify the specific adversary techniques and tactics that operationalize that threat. This pairing gives you both the vulnerability class and the attacker behavior together, which is what you need to reason through detection and mitigation realistically.

Work through each trust boundary individually rather than trying to enumerate threats against the system as a whole. Boundary-level analysis surfaces threats that get missed in system-level analysis, consistently.

Phase 3: STRIDE Analysis with LLM Extensions

Apply STRIDE to each component and data flow, then explicitly check for the three categories that classic STRIDE misses: Prompt Injection including indirect vectors, Context Poisoning especially relevant for agentic systems with persistent memory, and Unsafe Tool Use where agents misuse legitimate permissions through indirect injection.

Use the ASTRIDE extensions or an explicit OWASP overlay to fill these gaps. A threat that fits no STRIDE bucket can still be real.

Phase 4: Agentic-Specific Analysis

If your deployment includes agents, run a separate analysis against each agent's full action space. Map every tool it can invoke, every API it holds credentials for, every permission it carries. Ask directly: if this agent were compromised through indirect prompt injection, what is the maximum impact? What data could be exfiltrated? What actions could be triggered? What downstream systems could be affected?

This analysis tends to reveal over-provisioned permissions that can be reduced before deployment. That's a low-cost finding with meaningful risk reduction.

Phase 5: Risk Scoring and Prioritization

Score threats using DREAD or an equivalent framework, with defined rubrics applied consistently across the team. Include impact as a distinct dimension from likelihood: a low-likelihood threat with catastrophic impact warrants high-priority treatment. For agentic threats, weight the action-space factor explicitly, since the same injection technique carries substantially higher impact when the agent has broader tool access.

Cross-reference findings against your data exfiltration path analysis. Threats that touch the output boundary, where model output flows to systems that execute or render it, generally warrant elevated priority because the blast radius includes exfiltration to external endpoints.

Phase 6: Mitigation Mapping and Residual Risk Review

Map each prioritized threat to specific, assignable mitigations. ATLAS provides 32 mitigations in v5.1.0; the OWASP categories include production-grounded mitigation guidance. "Validate model outputs" is not a mitigation plan. "Implement a content security policy that prevents rendered model output from loading external resources" is.

After mapping mitigations, review residual risk explicitly. Some threats, particularly those rooted in the probabilistic nature of LLMs, cannot be fully mitigated. They can be reduced, monitored, and contained. Naming what remains open is more useful than pretending the mitigation list closes every gap. It's also what allows you to have an honest conversation with leadership about acceptable risk levels, which is increasingly a conversation that needs to happen and that most teams are underequipped to facilitate.

This isn't a one-time exercise. LLM applications evolve: new tools get added, RAG document stores grow, agent capabilities expand, new adversary techniques emerge. Revisit the threat model when the architecture changes. Check it periodically against current OWASP and ATLAS versions, because both frameworks update frequently enough that a model built against last year's editions has genuine gaps today.

The AI-related security incident numbers, 56% increase in 2025, nearly a third of organizations surveyed by Gartner reporting an attack on an AI application within twelve months, reflect what happens when generic frameworks meet a structurally novel attack surface. Not a single catastrophic failure, but quiet, category-level gaps accumulating until something goes wrong. The case for a structured, LLM-specific methodology is in that incident data. It's already there.