Overreliance on LLM Output as a Security Risk

The framing most organizations bring to this problem is already wrong. They treat it as a training issue: teach people to verify outputs, remind analysts to stay skeptical, circulate a policy memo, and call it managed. But what if the risk isn't primarily a behavioral one at all? It's architectural. The model generates fluent, confident, authoritative-sounding text whether or not the underlying claim is accurate. The pipeline accepts that output and routes it toward decisions. And the humans in the middle, shaped by well-documented cognitive dynamics, apply less scrutiny than the situation demands, not because they're careless, but because the system is designed, inadvertently, to suppress scrutiny.

OWASP's decision to rename their 2025 LLM Top 10 category from "Overreliance" to "Misinformation" signals exactly this reframe. The risk doesn't live in user behavior alone. It lives in the model as a generator and propagator of false information, and in the architecture that carries that information forward to consequential action. The rename is a standards body acknowledging that the vulnerability is systemic.

What follows is an attempt to trace the actual mechanism: three converging attack surfaces, all sharing overreliance as their root. Misinformation propagating through enterprise workflows. Hallucinated software dependencies weaponized by attackers. And the skilled security analysts who are supposed to catch these failures, gradually losing the judgment required to do so.

The Cognitive Machinery Behind Misplaced Trust

Before the attacker does anything, the cognitive setup is already in place.

To understand why this works, we must first look at the three biases that interact to produce overreliance, and how LLMs amplify all three in ways earlier automation didn't. Automation bias is the tendency to defer to an automated system's output even when your own judgment contradicts it, even when the system is visibly wrong. Complacency bias compounds it: once a tool has been reliable enough in the past, prior correct answers lower your guard for the next response. You've been right alongside it so many times that you stop checking as carefully. And then there's cognitive offloading, the deliberate reduction of mental effort by delegating reasoning to the tool, which, over time, atrophies the critical evaluation habit you'd need to catch errors when they finally appear.

Earlier automation didn't trigger these biases as forcefully, because earlier automation didn't mimic human reasoning. A spreadsheet formula is clearly mechanical. A database query returns structured data. But natural language output activates the same social trust heuristics you apply when evaluating another person. Fluency reads as competence. Confidence reads as correctness. These are heuristics that evolved for assessing humans, and they're being systematically misapplied to a stochastic text generator.

Then add what the field calls the "illusion of universal competence." Ask a human expert a question outside their domain, and they'll hedge, qualify, or decline. LLMs respond to nearly any prompt with apparent authority, masking domain-specific limits and inconsistencies that would, in a human, signal to you that you should look elsewhere. The model doesn't know what it doesn't know, and critically, it doesn't signal that uncertainty in a legible way.

Sycophantic training behavior makes this worse in a specific way. Models optimized for user approval are more likely to confirm your framing than contradict it. If you present a flawed premise in your prompt, the model is statistically inclined to build on it rather than push back. This means the errors most likely to propagate are the ones that align with what the user already believed, which are precisely the errors that feel least like errors.

The implication for security framing matters here: because the mechanism is cognitive and architectural, user training cannot fully close the gap. Telling analysts to be more skeptical is fighting the bias with willpower, which is a strategy that degrades under pressure, under fatigue, and under conditions of high alert volume. The vulnerability has to be addressed at the system and process layer, not just the human one.

The Reliability Numbers, and What They Actually Mean

The instinct is to frame hallucination as an edge case, a failure mode that shows up under unusual conditions or in poorly constructed prompts. The data suggests otherwise.

A 2025 evaluation of 40 AI models, using Artificial Analysis's AA-Omniscience benchmark, found that all but four models were more likely to give a confident, incorrect answer than a correct one on difficult questions. Read that again slowly. Confident wrongness is the statistical norm, not the outlier, across nearly the entire evaluated field. That raises an important question: if confident wrongness is the baseline, what does it mean to treat these outputs as reliable?

The enterprise exposure numbers compound this. According to 2024 findings, 47% of enterprise AI users admitted to making at least one major business decision based on hallucinated content. The downstream correction costs are tangible: 39% of AI-powered customer service implementations were pulled back or substantially reworked due to hallucination-related errors. In Q1 2025 alone, over 12,000 AI-generated articles were removed from online platforms for hallucinated content.

Per the Stanford AI Index Report 2025, documented AI safety incidents rose from 149 in 2023 to 233 in 2024, a 56.4% single-year increase, with hallucinations accounting for 38% of those incidents. And 76% of enterprises have now implemented human-in-the-loop processes specifically to catch hallucinations; most of them learned this by absorbing the cost of not having those processes. The 24% who haven't are carrying the greatest current exposure.

What the trend line implies is the part worth sitting with. Incident volume is growing faster than the enterprise adoption curve. Risk is not self-correcting as models mature. The assumption that better models will simply outgrow this problem is not supported by the trajectory.

When Hallucinated Output Reaches a Consequential Decision

There's a distinction worth drawing carefully here, between an accuracy failure and a security failure. An accuracy failure is a wrong answer the user could catch with minimal scrutiny. A security failure is a wrong answer that looks authoritative, reaches a downstream system or decision, and creates an exploitable or legally binding condition before anyone checks. The gap between those two failure modes is exactly what makes overreliance a formal risk.

The Air Canada case illustrates the low-stakes channel version. The airline's chatbot hallucinated a bereavement discount not in company policy. A customer relied on it, made a travel decision accordingly, and when Air Canada refused to honor the offer, a tribunal compelled them to do so. The hallucination was minor. The legal obligation it created was not. Harm occurred at the moment of customer action, not at the moment of generation, and no verification happened in between.

The legal citation cases represent a higher-stakes failure pathway. In November 2024, in Gauthier v. Goodyear Tire, an attorney submitted a brief containing nonexistent case citations and fabricated quotations from real cases, language indistinguishable in style from actual judicial writing. A Florida attorney, James Martin Paul, used hallucinated citations across eight separate legal matters, resulting in sanctions and dismissal without prejudice of four federal cases. The AI Hallucination Cases Database tracked 486 such incidents worldwide by 2025, 324 of them in U.S. courts. Early incidents tended to involve one to three hallucinated citations per brief; more recent cases involve significantly higher volumes per filing, as LLM use has deepened in legal practice.

The escalation pattern is its own signal. It suggests that as reliance increases, the frequency of error per instance increases, not decreases. But how does this affect our original promise that more experienced users would catch more errors? It doesn't. The pattern runs directly counter to that assumption.

The cybersecurity operations context takes this further. An AI system confidently recommending deletion of sensitive files, modification of system configurations, or disabling of firewall rules, executed through a privileged account, can enable identity-based attacks, lateral movement, or irreversible data loss. The gap between generation and action is where verification should happen. In automated or agentic systems, that gap does not exist at all.

Microsoft Copilot, a flagship enterprise product, produced factual errors in approximately 30% of responses to election-related questions during 2024 testing, including incorrect dates, wrong candidate information, and invented quotations. This isn't obscure tooling. It's the product millions of organizations are deploying as a productivity layer over sensitive workflows.

Slopsquatting: Predictable Hallucinations as an Attack Surface

The elegance of slopsquatting as an attack is that it requires no deception of the developer. It only requires deception of the model.

The mechanism works like this. An LLM code assistant, when asked to generate code that requires external libraries, will sometimes hallucinate package names that don't exist. An attacker pre-registers one of those names on npm, PyPI, or another public registry. A developer runs the AI-suggested install command, pulls down the malicious package, and the attacker has execution within the developer's environment. The developer did nothing wrong by their own standard operating procedure. They trusted the output.

A study presented at USENIX Security 2025, conducted by researchers from the University of Texas at San Antonio, the University of Oklahoma, and Virginia Tech, generated 2.23 million code samples across 16 models. Of those, 19.7% contained at least one hallucinated package name, and researchers identified over 205,000 unique fabricated package names. The model-class breakdown matters for procurement decisions: open-source models hallucinated packages at an average rate of 21.7%; commercial models averaged 5.2%; the CodeLlama family exceeded 33%; GPT-4 Turbo achieved the lowest measured rate in the study at 3.59%.

What makes slopsquatting categorically more dangerous than traditional typosquatting is predictability. Typosquatting depends on human keystroke errors, which are probabilistically random. Slopsquatting depends on a model's hallucination patterns, which are reproducible. When researchers re-ran identical prompts ten times, 43% of hallucinated package names reappeared on every single run; 58% appeared on more than one run; 61% appeared across multiple runs. An attacker who catalogs a model's hallucinations can pre-register those names with high confidence they'll be requested again.

Traditional defenses don't transfer. npm's collision-detection logic looks for names similar to existing packages, because typosquatting produces strings adjacent to real names. Hallucinated names are novel strings with no real-name collision to detect, so the existing protection mechanism has no surface to act on. The attack class is invisible to the defense designed for the analogous attack class.

Palo Alto's Unit 42 research extended this logic to web infrastructure, finding that LLMs can generate fictitious domains for web portals, API endpoints, or corporate services. The same hallucination vector moves from package registries to DNS-level infrastructure. The scope is broader than any single registry or ecosystem.

The deeper implication is the one organizations most want to avoid confronting: slopsquatting is not a model bug awaiting a patch. It is a feature of probabilistic text generation being systematically exploited. Reducing hallucination rates reduces exposure. It does not eliminate the attack class.

AI-Generated Code as a Vulnerability Pipeline

Nearly 70% of organizations estimated in 2024 that over 40% of their code was AI-generated. 92% of security leaders expressed concern about the security implications. Those two numbers in proximity describe an organization that has adopted the risk faster than it has built the capacity to manage it.

The 2025 Veracode GenAI Code Security Report found that AI-generated code introduced security vulnerabilities in 45% of 80 curated coding tasks evaluated across more than 100 LLMs, with models choosing an insecure implementation over a secure one at effectively the same rate. The weakness classes most commonly introduced, including buffer overflows, hard-coded credentials, SQL injection, cryptographic misuse, and path traversal, are established CWE categories. Not novel attack classes. Detectable with existing tooling, if someone runs it.

That qualifier does a lot of work. The velocity of the problem makes the "if someone runs it" condition increasingly difficult to satisfy. By mid-2025, AI-generated code was introducing over 10,000 new security findings per month, a tenfold increase from December 2024. The backlog is growing faster than most security teams can triage it.

The feedback loop that compounds over time is the structural concern. Insecure AI-generated code ships to production. That code enters public repositories. It eventually enters training corpora. Models learn to reproduce insecure patterns at higher frequency. What previously spread through manual copy-paste now propagates across thousands of codebases in a timeframe no human review process can match.

Developers who accept AI code suggestions without security review are not making a skill error. They are behaving exactly as automation bias predicts. Exhorting them to be more careful is not an architectural control. Mandatory review gates are.

How Automation Bias Degrades the Analysts Meant to Catch These Failures

The reasonable assumption is that trained security professionals are partially immune to the cognitive dynamics that affect general users. The data indicates partial immunity, at most. One might argue that professional training and skepticism create a meaningful buffer — but does the evidence actually support that?

A 2025 study published in ACM's Digital Threats: Research and Practice, based on interviews with 19 cybersecurity professionals, found automation bias influencing 47% of analysts' trust in AI-driven security tools, and confirmation bias affecting 37%. These aren't naive users. They're professionals whose job is to evaluate these systems critically.

65% of analysts in that study expressed skepticism toward AI alerts, yet 79% favor hybrid human-AI models over full automation. The profession has diagnosed the right instinct. The structural conditions that produce automation bias persist regardless of that expressed preference, because bias is not a preference, it's a tendency that operates below deliberate choice.

The false positive dynamic creates a specific paradox. AI hallucinations in security tooling can fabricate non-existent vulnerabilities and misidentify threat intelligence. Repeated false positives generate alert fatigue: analysts become desensitized to warnings generally, including real ones. Overreliance, trusting AI recommendations to act, and underreliance, dismissing AI alerts as noise, are both downstream of the same broken calibration. The failure modes look opposite; they share a root.

Skill erosion is the lagging indicator that takes longest to notice. Analysts who outsource detection reasoning to AI tools over time lose the practiced judgment required to evaluate AI output critically. The very competency needed to catch AI errors atrophies through disuse. This is not hypothetical; it's the documented trajectory of automation dependency in aviation, medicine, and financial analysis, now replaying in security operations.

When no institutional mechanism exists to identify analysts who fail to critically evaluate AI outputs, individual automation bias aggregates into organizational blind spots. A single flawed AI recommendation, executed through a privileged account, can trigger lateral movement or irreversible data loss before any human review occurs.

What the Regulatory Frameworks Now Require

The legal and standards frameworks have caught up faster than many compliance teams realize.

The EU AI Act's Article 14 on Human Oversight requires that high-risk AI systems be designed with interfaces enabling the humans responsible for oversight to understand system capabilities and limitations, detect and intervene on failures, and override or interrupt system operation. For covered systems, overreliance is not merely inadvisable. It is non-compliant.

OWASP's LLM Top 10 2025, in its "Misinformation" category, recommends output validation, retrieval-augmented generation with verified sources, and explicit human review checkpoints before AI-generated content reaches consequential decisions or external communications. The standard is specific about where the checkpoint must sit.

NIST's AI Risk Management Framework and associated U.S. executive guidance frame human oversight as a core "Govern" and "Manage" function, with measurable thresholds for when AI output requires human review before action. The legal force differs from the EU Act, but the directional requirement is consistent.

It is also worth considering the compliance gap most organizations currently face: it is architectural, not policy-level. Governance frameworks were written assuming humans review AI outputs before action. Actual deployment, particularly agentic AI systems that can execute on their own outputs, increasingly automates the action step. The regulatory text assumes the review happens. The architecture often ensures it doesn't.

Vendor risk adds a layer that third-party risk management programs are only beginning to address. When an enterprise deploys a vendor's AI product, the enterprise typically inherits the compliance obligation for human oversight, even though the model's hallucination rate and design choices were set by the vendor. The accountability follows the deployment, not the model's origin.

Controls That Address the Mechanism, Not the Symptom

The principle organizing the control set is straightforward: controls must interrupt the pathway from generation to consequential action. They cannot rely on asking users to sustain greater skepticism, because skepticism degrades under automation bias, alert fatigue, deadline pressure, and workload. Structural gates do not degrade the same way.

What this looks like in practice begins with the generation layer: retrieval-augmented generation architectures that ground model outputs in verified, citable sources before presenting them to users reduce hallucination rates in domain-specific applications, and they produce outputs that can be audited against the source material. This isn't a cure; it's a meaningful constraint on the failure surface.

At the workflow layer, the control is mandatory review checkpoints positioned between AI output and consequential action. Not optional review. Not a disclaimer encouraging verification. A hard gate that requires a human to affirm before the output proceeds to execution. For code, this means security scanning before AI-generated code reaches any environment where it executes, with tooling calibrated for the CWE categories AI models most commonly introduce. For legal, clinical, or financial outputs, it means structured sign-off workflows, not trust on delivery.

For security operations specifically, the answer to automation bias is not removing AI from the analyst's workflow; it's designing the workflow so the analyst's critical evaluation is structurally required before action executes. This means alert designs that surface supporting evidence alongside recommendations, requiring analysts to engage with the reasoning rather than simply accept or dismiss the conclusion.

Monitoring for overreliance as a measurable operational metric closes the institutional visibility gap. Tracking the percentage of AI recommendations acted on without modification or review, the volume of AI-generated code that clears a review gate unchanged, and the ratio of analyst override decisions to AI-recommended actions gives organizations something they currently mostly lack: a leading indicator of where automation bias is accumulating before an incident crystallizes it.

The harder cultural shift is treating overreliance as a formal risk category with ownership, metrics, and accountability, rather than a user behavior problem that the next training cycle will address. The standards bodies have already made this move. OWASP's rename is one signal. The EU AI Act's specificity on human oversight mechanisms is another. The incident trajectory in legal proceedings, customer-facing systems, and security operations is a third.

The question for any organization running AI in consequential workflows is not whether their model hallucinates. It does. The question is whether anything in their architecture stands between that hallucination and a decision, a legal filing, a deployed package, or an executed system command. Right now, for a significant portion of enterprise deployments, the honest answer is: not reliably enough.