LLM Security Testing Methods and Tools

Before any testing method makes sense, you need a shared taxonomy of what failure actually looks like in production. The OWASP Top 10 for LLM Applications, updated to version 2.0 in 2025, is that taxonomy. Without it, you are probing without a target, which is less useful than it sounds and more common than anyone admits.

The ten categories reward careful attention. The ranking shifts between versions tell their own story about where real pressure is building.

Prompt Injection holds the top position. An attacker crafts input that overrides or bypasses developer instructions, steering the model toward outputs or actions it was explicitly designed to avoid. Every section that follows orbits this one.

Sensitive Information Disclosure jumped from sixth to second place in 2025, driven by demonstrated training-data extraction techniques. Researchers have shown, repeatedly, that LLMs can be induced to reproduce verbatim passages from their training corpora, including personal data and proprietary content. The jump in ranking reflects real incidents, not theoretical concern.

Supply Chain risk covers the full dependency chain: base models, fine-tuned variants, datasets, plugins, orchestration frameworks, vector databases. Every third-party component is a potential ingress point, and most organizations cannot fully enumerate what they are pulling in.

Data and Model Poisoning addresses manipulation of training data, feedback loops, or the retrieval knowledge bases that power RAG systems. The attacker does not need access to the model at inference time; they need access to what the model learns from. That is a fundamentally different threat model than most security teams are calibrated for.

Improper Output Handling is about what happens downstream. If a system consumes raw LLM output without sanitization and passes it to a browser renderer, a code executor, or a database query, the model becomes a relay for injection attacks into adjacent systems. The LLM is not the target; it is the vector.

Excessive Agency reflects what happens when models are granted tool access and autonomy beyond the minimum necessary. In agentic systems, the blast radius of any single failure expands dramatically.

System Prompt Leakage is new in 2025, added after real incidents demonstrated that system prompts could be extracted through adversarial conversation, exposing internal rules, filtering logic, and permission structures the developer intended to keep confidential.

Vector and Embedding Weaknesses earned its place as more than half of enterprises now rely on retrieval-augmented generation rather than fine-tuning. Poisoned retrievals, cross-tenant data exposure, and adversarial manipulation of embedding similarity are the operative failure modes, and they surface at inference time, often long after the poisoning occurred.

Misinformation, renamed from "Overreliance" in the 2025 revision, sharpens the focus to the model actively generating and propagating false information, not merely users trusting it too readily.

Unbounded Consumption covers token exhaustion, denial-of-wallet attacks, and unauthorized model replication. These become significant at scale in ways that are easy to underestimate during early deployment.

In late 2025, OWASP also released a separate Top 10 specifically for agentic AI systems, recognizing that applications where LLMs autonomously plan and execute multi-step tasks represent a distinct risk landscape from standalone chatbots. If you are deploying agents, that list runs in parallel with the standard Top 10, not as a replacement.

A testing programme needs coverage across categories, not just around the headline vulnerability. The methods that follow each target a different slice of this map.

Prompt injection as the central testing challenge

Prompt injection deserves its own treatment because it is simultaneously the most prevalent vulnerability class, the hardest to remediate fully, and the one that amplifies every other failure mode when a model has real capabilities attached to it.

Two attack types require distinct testing approaches. Direct injection is what most people picture first: an attacker types malicious instructions into the interface. Visible, reproducible, relatively tractable to test systematically. Indirect injection is considerably more dangerous and considerably harder to test. The malicious instructions are hidden inside external content the model retrieves and processes, a web page, a PDF, an email, a document in a connected drive. The user never sees the attack. The model does.

Attack success rates against hardened configurations, applying the best available defenses from major providers, range from 50% to 84% depending on system configuration. CVE-2025-53773, a critical vulnerability in GitHub Copilot rated 9.6 on the CVSS scale, enabled remote code execution through prompt injection. Comparable critical CVEs appeared in other widely deployed coding assistants during the same period. These are not research curiosities. And if success rates remain that high against hardened configurations, that raises an important question: is a testing programme framed around achieving a binary safe verdict built on the wrong premise entirely?

The agentic amplification problem is what makes this category so consequential. When a model can browse the web, execute code, query databases, and call APIs, a successful injection does not produce a bad text output; it produces a cascade of unauthorized tool calls. The failure propagates outward through every capability the model can reach.

The UK's National Cyber Security Centre assessed in December 2025 that prompt injection will resist full mitigation the way SQL injection eventually was addressed, characterizing LLMs as "inherently confusable deputies." That characterization reorients the testing goal toward something more honest: you are measuring attack success rates across configurations and treating defense-in-depth as the realistic target. A programme that still runs a pass/fail check on this category will eventually cost someone.

Reported prompt injection incidents increased 340% year-over-year between 2024 and 2025. If your programme treats this as a low-frequency edge case, the calibration is off.

Manual red teaming: what it covers and where automated tools cannot substitute for it

Manual red teaming is adversarial testing conducted by a human who approaches the application as an attacker. Not as a quality assessor. Not as a code reviewer. As someone trying to make the system do something it was designed not to do. That orientation changes what the tester looks for and how they probe, and no amount of scripting fully replicates it.

What manual red teaming uniquely catches is the multi-turn crescendo attack. Over the course of five, ten, sometimes twenty exchanges, a skilled tester gradually steers a conversation toward a target output, exploiting the model's context window and its tendency to accommodate conversational momentum. Detection systems that evaluate messages in isolation, which describes most automated safety layers, have no visibility into this kind of accumulated context manipulation.

NVIDIA's AI Red Team assessed in 2025 that automated tools catch between 40% and 60% of vulnerabilities. The remainder requires manual expert testing. That gap is not a temporary limitation of current tooling. It reflects the structural difference between probing known attack patterns and exercising adversarial judgment, two things that are not the same and cannot be made equivalent by adding more probes to a library.

A well-structured engagement typically begins with a reconnaissance phase of one to two days. The tester determines the architecture, maps all data entry points, identifies available tools and privilege levels, and attempts to extract the system prompt. That last step is itself a live test of LLM07; how the model responds to extraction attempts tells you something meaningful about its configuration before the active probing even starts. I have run sessions where a system appeared robustly configured right up until the system prompt surfaced in the first twenty minutes. What you learn from that moment recalibrates everything that follows.

Results from manual red teaming are reported as statistical distributions: attack success rates across categories like prompt injection, jailbreaking, data extraction, and behavior manipulation. Not a pass/fail verdict. If you receive one of those from an engagement, ask the provider what it means, because it almost certainly does not mean what it implies.

Manual red teaming is expensive, slow to scale, and non-reproducible across model updates. When the underlying model changes, the findings from the last engagement do not automatically carry forward. That is precisely what makes the automated tools necessary, complements with distinct ceilings, not substitutes for each other.

Automated fuzzing for LLMs: how it works and what Garak covers

Traditional fuzzing mutates binary inputs until a system crashes. LLM fuzzing operates on a different premise entirely: it generates semantically varied adversarial prompts to probe behavioral failures, hallucination, policy bypass, data leakage, toxicity. The failure mode is not a crash; it is an output the model was supposed to suppress. That distinction shapes everything about how these tools are built.

Evolutionary fuzzing, as implemented in research frameworks like LLM-Fuzzer, generates prompt mutations from a seed set using evolutionary algorithms. Each successful mutation becomes the seed for the next generation of payloads. The attack improves as it runs, learning in real time what the model is susceptible to. Watching a well-seeded evolutionary fuzzer work across a model surface is genuinely clarifying about where the soft spots are.

Garak, developed by NVIDIA, is the primary open-source tool for broad-surface LLM scanning. The comparison to Nessus is apt: it is a breadth scanner designed to map the surface before you commit resources to targeted investigation.

Its probe library covers more than 37 modules across major model APIs. Relevant categories for practitioners include promptinject for direct injection variants, leakreplay for training-data extraction attempts, encoding probes for base64, ROT13, and leet-speak bypass attempts, dan probes for jailbreak persona attacks, continuation probes that elicit harmful content through story framing, suffix probes deploying universal adversarial suffixes, and misleading probes designed to induce hallucination.

The appropriate use for Garak is initial surface mapping, regression checks after model updates, and triage before committing manual red team hours to a target. Run it first, let it identify where the soft surfaces are, then route those findings to deeper investigation. Using it as a substitute for that deeper investigation is the mistake.

Garak operates from a static probe library. It cannot adapt to the target model's live responses, and vulnerabilities that only emerge through conversational context are invisible to it. That is not a criticism of the tool; it is a description of its ceiling, and knowing where that ceiling sits determines how you use it.

Adaptive multi-turn attack simulation with PyRIT

PyRIT, Microsoft's open-source red teaming framework, was built and refined through more than a hundred internal operations before it was open-sourced, including testing against Phi-3 and Copilot. That provenance matters. It was not designed in theory; it was built by practitioners running real engagements who needed something that could do what static tools could not.

The core architectural distinction from Garak: instead of a static probe library, PyRIT uses an orchestrator LLM that acts as the attacker. It generates adversarial prompts, observes the target model's responses, and refines its approach based on what it sees. The attack adapts in real time, which is a qualitatively different capability from anything probe-based. But what if the attacker's model is more capable than the defender's? That is not a hypothetical edge case — it is a live variable in any adaptive campaign.

This is what enables meaningful multi-turn simulation. Crescendo attacks require an attacker that holds context and adjusts strategy across many exchanges. A static probe library cannot do that. An orchestrator LLM can, and teams who have used both tend to stop treating them as interchangeable fairly quickly.

In April 2025, Microsoft released the AI Red Teaming Agent within PyRIT for automated orchestration of structured attack campaigns, with Azure AI Foundry integration. The attack library covers prompt injection, jailbreaking, and content safety testing across these adaptive conversational patterns.

One development that changes the stakes: in 2026, large reasoning models demonstrated they could independently plan and execute multi-turn jailbreak strategies against other AI models. The adaptive attack pattern PyRIT pioneered as a defender's testing tool is available to adversaries without a testing framework at all. The gap between what defenders can automate and what attackers can automate is narrowing, which has obvious implications for how seriously organizations treat the timeline for building these programmes.

PyRIT's limitation is the inverse of Garak's. It is depth-focused, not breadth-focused. Running sustained adaptive campaigns against every surface is resource-intensive, so the correct sequencing is to use Garak to identify which surfaces warrant that investment, then bring PyRIT to bear on those specific areas. Treating them as competitors rather than as a sequence is a resource allocation mistake.

CI/CD integration and regression testing with Promptfoo

Here is a scenario that plays out more than practitioners tend to admit publicly. A model passes a red team engagement. The security team signs off. Three weeks later, the underlying model is updated, or the system prompt is revised, or a new plugin is added. The safety layer that passed testing no longer behaves the way it did when it was tested, and nobody catches it because nobody ran the tests again.

Point-in-time testing does not protect production. The red team report is a snapshot of a system that has since changed, not a certificate of ongoing safety. Every model update that ships without safety regression testing is essentially untested from a behavioral standpoint, regardless of what the previous engagement found. It is also worth considering what this means organizationally: if the team that ran the original engagement has moved on, there may be no one who fully understands what the prior tests were actually measuring.

Promptfoo addresses this gap. It is a CLI-based tool built CI/CD-first, designed for configuration-driven regression testing of safety layers within a deployment pipeline. Tests are defined as code, versioned, and reproducible, which means they actually run rather than depending on someone remembering to run them manually.

Its key operational capabilities include side-by-side model comparison, running the same attack payloads across multiple model versions or configurations simultaneously so you can observe behavioral drift before it ships. Its dedicated red team mode automates safety regression across the OWASP Top 10 categories on a scheduled or triggered basis.

The pipeline practices this enables are standard DevSecOps principles applied to a novel attack surface: automated safety evaluations on every model or prompt update, prompt scanning for malicious patterns at ingestion, output monitoring for policy violations, audit logs maintained for regulatory review. None of these ideas are novel. The novelty is applying them consistently to LLM deployments, which most organizations are not yet doing.

Promptfoo operates against known attack patterns. It will not generate the novel adversarial reasoning that a skilled human red teamer or an adaptive LLM attacker can produce. Its value is in ensuring that known-bad patterns do not regress after changes, not in discovering new failure modes. Those are two distinct goals, and conflating them creates a false sense of coverage.

Testing the agentic and RAG attack surface specifically

Standalone chatbot security and agentic security are not the same problem. When a model can browse the web, execute code, query databases, and call APIs, a single prompt injection no longer produces a bad text output; it produces unauthorized action cascading through tool calls. The character of the failure changes entirely, and teams whose testing programmes were designed around the chatbot model are ignoring precisely the surface that matters most in their current deployment.

The agentic attack surface requires explicit testing at two layers.

At the tool-call layer, the relevant attack vectors are tool-call hijacking, indirect prompt injection through external data sources, MCP server poisoning, cross-agent injection in multi-agent pipelines, and privilege escalation through chained tool calls. Each of these requires the tester to think about the full action graph of what the model can do, not just what it will say. In practice, most security teams have not yet built that mental model, partly because the frameworks for agentic deployment are newer than the security intuitions practitioners bring to them.

At the runtime layer, the targets are denial-of-wallet attacks through token exhaustion, context window manipulation, guardrail evasion through multi-step routing, and system prompt extraction. Excessive Agency, LLM06, deserves particular scrutiny: the majority of deployed agents are scoped with permissions well beyond the minimum necessary, and testing what an attacker can accomplish with those excess permissions is a foundational check that is frequently skipped.

OWASP's separate Top 10 for agentic AI systems, released in late 2025, provides the classification framework for this surface, running in parallel with the standard LLM Top 10.

RAG systems introduce their own distinct testing requirements. More than half of enterprise LLM deployments rely on retrieval-augmented generation rather than fine-tuning, which means vector database and embedding weaknesses, LLM08, are in scope for the majority of production systems.

Vector database poisoning tests whether injecting malicious content into the knowledge base causes it to be retrieved and incorporated into responses during legitimate queries. The attack happens at ingestion time; the damage surfaces at inference time, potentially long after, through queries that look completely benign. Cross-tenant data exposure testing validates whether access controls on vector stores are sufficient to prevent sensitive data from crossing organizational boundaries when multiple tenants share an index. Embedding model manipulation tests whether adversarial inputs can skew similarity results at the embedding layer, causing the retrieval system to surface attacker-controlled content preferentially.

None of this is exotic. If your organization is among the majority relying on retrieval-augmented generation and your testing programme does not include these checks, you have a structural gap regardless of how thorough the rest of your coverage is. The gap is not speculative; it is the predictable consequence of deploying a surface you have not tested.

LLM security testing is not one method applied uniformly. It is a set of distinct methods, each designed for a different threat model, each with a different ceiling, and the practitioners building programmes that actually hold up are the ones who understand where each method's coverage ends, not just where it begins.