NIST AI RMF Applied to LLM Deployments

GOVERN, MAP, MEASURE, MANAGE: that's the vocabulary NIST's AI Risk Management Framework hands you. What it doesn't hand you is a way to run that vocabulary against a large language model that hallucinates, leaks data, or gets hijacked by a prompt buried inside a document it just retrieved. This piece walks through each of the four functions and asks the same question at every stop: what does this actually look like once an LLM is live and taking requests? The honest answer, up front: most organizations stop at documentation and call it done, and that's exactly where the framework starts to fail them.

Stackcurve frames the gap well: having "mapped to NIST AI RMF" is like having read the fire code. Having the framework actually running is like having sprinklers tested every quarter. One is paperwork. The other puts out fires.

Stackcurve's telling of an SEC inquiry makes the distinction concrete. A publicly traded tech company disclosed that it had adopted the NIST AI RMF. Six months later, a generative AI feature started producing systematically wrong outputs, and a class of customers got hit by it. When the SEC came asking questions, the inquiry zeroed in on three things: no ongoing check on that feature's output quality, no defined performance thresholds, and no incident response that actually kicked in. The framework adoption claim didn't matter much once regulators looked for the plumbing behind it.

That's the pattern to sit with. Governance existed on paper. Nothing existed at runtime, and that's the gap that actually gets you in trouble.

It isn't rare, either. Industry surveys consistently find that organizations adopting AI tools far outnumber those with a real governance structure behind that use. Stackcurve's 2026 CURVE report, drawn from 41 enterprise respondents, breaks the maturity curve down further: 58% sit at Level 2, having mapped and written down their AI use and stopped there. Another 22% are at Level 3, with policies on paper but no ongoing measurement behind them. Only 12% have reached Level 4, where controls actually run day to day, and just 8% are at Level 5, improving continuously.

Why does this gap widen specifically for LLMs? Hallucination, prompt injection, data exfiltration, model drift: none of these are risks you assess once and file away. They're runtime phenomena. A policy reviewed once a year can't catch a prompt injection attack that happened Tuesday afternoon. NIST's own generative AI profile, AI 600-1, published in July 2024, exists precisely because the base framework didn't fully see these dynamics coming. The gap between having a policy and actually running one is bigger for generative AI than it ever was for traditional machine learning. The rest of this piece is about closing it, function by function.

What NIST AI RMF 1.0 and NIST AI 600-1 actually require, and what they leave to the implementer

AI RMF 1.0 came out January 26, 2023, as voluntary guidance developed under the National Artificial Intelligence Initiative Act of 2020. More than 240 organizations, spanning industry, academia, civil society, and government, fed into its drafting. It's built around four functions, broken into 19 categories and 72 subcategories, and written on purpose to work across any industry and any type of AI system. That neutrality is a feature for a framework meant to apply broadly. It's also exactly why it doesn't tell you what to do about an LLM specifically.

One structural detail matters here: GOVERN sits at the organizational level, covering the whole enterprise. MAP, MEASURE, and MANAGE apply system by system, to each individual AI deployment. That split shapes everything that follows.

NIST AI 600-1, the Generative Artificial Intelligence Profile, filled part of the gap. Published in final form on July 26, 2024, in response to Executive Order 14110 from October 2023, it maps the same four functions onto 12 risk categories specific to generative AI: CBRN Information or Capabilities, Confabulation, Dangerous and Violent or Hateful Content, Data Privacy, Environmental Impacts, Harmful Bias and Homogenization, Human-AI Configuration, Information Integrity, Information Security, Intellectual Property, Obscene or Degrading Content, and Value Chain and Component Integration. It comes with a substantial set of suggested actions sorted by function and category, a real step up in specificity from the base framework.

But a suggested action still isn't implementation code. The profile tells you to address prompt injection under Information Security. It doesn't tell you which test suite to run, what threshold counts as a pass, or who gets paged when the threshold breaks. Tooling, measurement method, enforcement mechanism: all of that is left to whoever's actually running the system, and that's exactly where most programs quietly run out of road.

The regulatory weight varies depending on who's asking. For federal agencies, EO 14110 makes the framework effectively mandatory. Colorado's SB 26-189, replacing the state's original 2024 AI Act, requires disclosure and human review of automated decisions once it takes effect January 1, 2027, with civil penalties running up to $20,000 per violation, though it did not carry forward certain AI governance incentives the earlier law had included. Organizations under the EU AI Act's reach often use the NIST framework as an internal operating model, mapped over to the conformity assessment work the EU law actually requires.

Across every one of these regimes, the framework answers what to achieve. It leaves how open. Everything from here forward is about that how, specifically for LLMs.

GOVERN: building accountability structures that can actually enforce policy at inference time

GOVERN is the only function that spans the whole organization rather than a single AI system. Its job is setting policy, drawing accountability lines, and defining who has the authority to make risk calls. Subcategory GOVERN 1.1 requires that AI risk policies exist. It says nothing about whether anyone follows them, and that's exactly where most programs quietly stall out.

A governance charter sitting in a compliance folder proves nothing, per request, about whether policy got enforced the moment a user asked an LLM to summarize a customer's account history. Paper doesn't intercept API calls. It never has.

Industry research consistently finds that clear executive accountability for AI governance is the exception, not the rule, with board-level oversight rarer still. When accountability doesn't sit clearly at the top, the gaps in enforcement don't stay at the top either. They cascade down to the engineer deciding whether to ship a feature on schedule or wait for a review nobody's clearly assigned to run.

For LLMs specifically, GOVERN needs to answer a few pointed questions. Who owns risk decisions for each deployment, including the ones built on vendor models, third-party plugins, and agent frameworks the company didn't build in-house? Is there an intake process that forces every new AI vendor component through a governance review before it goes live? Are decision rights mapped out explicitly, meaning who approves a customer-facing feature that summarizes personal data, who can override that approval, who gets looped in when something goes wrong? And has the organization's risk appetite been written in terms that actually connect to LLM behavior, covering an acceptable confabulation rate, a list of banned output categories, and a defined scope of what data the model can touch?

Real GOVERN looks different from decorative GOVERN, and the difference shows up in the artifacts. A charter that names the specific LLM systems in production, not AI in the abstract. A third-party AI inventory updated as vendors change, not reviewed once a year. Roles tied to named people, not job titles that rotate. Acceptable-use policies with criteria you can actually check against.

The common failure mode is a charter so generic it could describe any company doing anything. "We manage AI risk responsibly" isn't a control. "This retrieval system may not pull documents outside data classification level X" is. GOVERN done well tells MAP exactly what systems exist and who's accountable for them, which is the foundation MAP needs to go find and characterize the risk in the first place.

MAP: finding every LLM deployment and characterizing what it can actually do to the organization

MAP's job is visibility: where is AI deployed, what risk does each deployment carry, and who gets affected if something goes wrong. Everything downstream, measurement and management both, depends on MAP being complete.

LLMs don't enter an organization through one door, and that's the inventory problem in a sentence. They come in through direct API integrations, through SaaS products that quietly embed a chatbot or summarizer, through developer tools and browser extensions, and increasingly through autonomous agents that string several of these together. Most of that entry happens before any formal review ever sees it.

Industry research consistently shows that formal AI incident-tracking capabilities remain rare, even in heavily regulated industries. An organization can't map what it hasn't catalogued, and most haven't built the tooling to catalogue it in the first place.

A MAP-complete LLM inventory needs to record, for every system: the foundation model's identity and version (sometimes called a model bill of materials, alongside a data bill of materials), what data it can reach at runtime (which corpora, which APIs, which retrieval index), where its outputs actually go (logs, a user interface, a downstream automated pipeline, a third party), who's affected by its decisions (employees, customers, regulated populations), and every plugin, connector, or vector store bolted onto it from outside.

Retrieval-augmented generation systems need extra scrutiny here. RAG blurs the line between data and instruction: a document sitting in the knowledge base can carry a hidden instruction that the model treats as a command, a classic indirect prompt injection. That risk doesn't show up if the inventory only lists which model version is running. It shows up only when someone maps the retrieval pipeline itself, document by document, connector by connector.

Value Chain and Component Integration, one of the 12 categories in AI 600-1, earns its place here too. A vendor-assembled AI stack brings risk at every layer it's built from, and the organization deploying it inherits that risk whether or not anyone assessed it. The MAP artifacts that matter include a system inventory with model and data BOMs attached, risk context documentation completed per system, a stakeholder impact map, and a register of every third-party component in play. MAP completeness decides whether MEASURE, the next function, tests the real risk surface or a cleaned-up version of it.

MEASURE: defining LLM-specific metrics and the infrastructure to collect them continuously

NIST describes MEASURE as evaluating AI risk through testing and ongoing monitoring of deployed systems. That word "continuously" carries real weight, and it's exactly the word that documentation-only programs skip right past.

The most common failure here is simple: organizations finish GOVERN and MAP on paper, then stall at MEASURE because they don't have the data pipeline to score risk on an ongoing basis. Writing a policy is cheap. Building a system that grades every production output against a threshold, every single day, is not.

A few metrics belong in every LLM MEASURE program. Prompt injection resilience needs a structured, adversarial test suite covering direct injection, indirect injection through documents pulled into a RAG system, and injection through tool outputs in agentic setups. OWASP's Top 10 for LLM Applications ranks prompt injection as risk LLM01:2025, the number one item on the list, which means it belongs in the baseline test protocol, not treated as an optional extra.

Confabulation rate matters just as much. Reporting from 2024 and 2025 puts enterprise LLM hallucination rates anywhere from under 1% in the best-performing systems to over 25% in some widely deployed models. That spread turns model selection, and how tightly a RAG pipeline grounds its answers in retrieved text, into a governance decision with a number attached to it, not just an engineering preference. Set the threshold for an acceptable rate before launch, then check against it continuously after.

Data leakage exposure needs its own metric: how often do outputs contain training data, personal information, or retrieved content the requesting user wasn't authorized to see? For RAG systems specifically, track a grounding number separately, the share of outputs faithfully sourced from retrieved context versus the share the model just made up. There's a quieter metric too: latency and cost as a stand-in for whether guardrails are actually working. A safety layer that slows response time enough creates real pressure, somewhere down the line, to quietly turn it off. Measuring the cost of the safety layer matters as much as measuring the quality of the output it protects.

Cloud deployment adds a wrinkle. MEASURE needs access to model behavior data, training data provenance, and runtime performance numbers, and cloud providers typically expose only part of that through their built-in observability tools. Weigh that constraint when choosing a deployment architecture, not after the fact.

A structural problem persists across the industry: standardized evaluation for LLM responsibility is thin, and leading developers test against different benchmarks. Cross-model comparison suffers as a result, which makes an organization's own internal benchmark consistency more important, not less. If outside benchmarks won't agree with each other, the internal one has to hold steady on its own.

The artifact that ties MEASURE together is the TEVV plan, short for test, evaluation, verification, and validation. It defines the test protocols, sets release thresholds, specifies how often monitoring runs, and documents the tooling generating all that telemetry. OWASP's LLM Top 10 plugs directly into MEASURE and MANAGE both, giving the security side of this work a concrete, adopted taxonomy to test against rather than one invented from scratch. MEASURE is what generates the signal, a breached threshold, a detected incident, drift in output quality, that MANAGE then has to act on.

MANAGE: closing the loop from detected risk to executed treatment for LLM-specific threats

MANAGE covers risk treatment, incident response, and corrective action. Subcategory MANAGE 1.1 requires that treatment plans exist. Like GOVERN 1.1, it doesn't require anyone actually execute them, and that gap is what running the framework operationally is meant to close.

The same 2025 AI Governance Survey cited earlier found fewer than 20% of organizations, again even in regulated sectors, have a dedicated AI incident reporting tool. Most companies can name their AI risks. Far fewer have built anything that responds once one of those risks shows up in production, at 2am, in front of customers.

A handful of playbooks need to exist and get tested before they're needed, not written up after an incident already happened. A prompt injection response playbook needs a trigger (an injection pattern flagged in telemetry), a containment step (blocking the query or killing the session), an escalation path, and a post-incident review. A confabulation playbook needs to define what happens when the hallucination rate set back in MEASURE gets crossed: who reviews it, who has the authority to pull the feature. A data exfiltration playbook needs a severity scale, deciding when an output gets automatically redacted, when it goes to a human reviewer, and when the session just gets cut off.

Model drift deserves its own line item. Vendors update models on their own schedule, sometimes without telling anyone, and that update can shift behavior in ways nobody downstream asked for. MANAGE needs a standing process to re-test against MEASURE's thresholds every time the underlying model changes beneath a deployment. And when a vendor's own LLM component is the source of an incident, the playbook needs to say, in advance, how to invoke that vendor's response process, what evidence to preserve, and how to tell affected customers what happened.

The evidence trail matters here too: runtime logs, telemetry exports, prompt and response records handled within privacy limits, escalation notes, and the remediation decisions that came out of it all. These serve two purposes at once, operational learning and regulatory documentation, and they only exist if someone built the logging pipeline to capture them in the first place.

The scale of the problem is growing fast. The Stanford AI Index counted 233 AI-related incidents reported in 2024, a 56.4% jump over 2023 and a 26-fold increase since 2012. An incident response plan nobody has rehearsed is a document, not a control. It sits there waiting to get tested for the first time under the worst possible conditions. MANAGE is also where the loop closes back on itself: treatment decisions feed back into MAP, updating the risk register, and into MEASURE, revising thresholds or adding new tests based on what the incident revealed.

What NIST AI 600-1's 12 risk categories add to each function in practice

The the suggested actions catalogued in AI 600-1, sorted across the 12 categories and mapped to all four functions, save an organization from having to invent an LLM threat taxonomy from scratch. That's the real value of the profile: it hands enterprises a starting vocabulary for risks that didn't have a standard name before generative AI made them common.

Information Security carries the most weight for most enterprise deployers, since it covers prompt injection, indirect injection through retrieved content, and supply-chain attacks against the tools an agent has access to. It lands squarely in MEASURE's evaluation protocols and MANAGE's control set, and lines up closely with OWASP's LLM01:2025 ranking of prompt injection as the top risk in that taxonomy. A March 2025 update from NIST expanded this category's coverage to include vulnerabilities specific to autonomous AI agents, reflecting how fast agentic systems have moved from experimental to deployed.

Confabulation is the category that turns hallucination from a known quirk of language models into something with a measurable threshold and an owner. It sits mainly in MEASURE, where the rate gets tracked, and in MANAGE, where crossing that rate triggers a defined response. Data Privacy plugs into MAP, since it depends on knowing exactly what data a model can reach, and into MEASURE, where leakage rate gets tracked as an ongoing number rather than a one-time audit finding.

Value Chain and Component Integration, discussed earlier under MAP, has consequences for MANAGE too: it's the category that forces an organization to plan for an incident caused by someone else's model or someone else's plugin, not just its own code. Human-AI Configuration touches GOVERN directly, since it's the category concerned with how much authority a human keeps over an AI system's output before that output reaches a customer or a regulated decision.

None of these categories replace the work of the four functions. What they give that work is a name, a place to sit, and a body of suggested actions to build from. That's the real difference between inventing an LLM risk framework from nothing and adapting a profile federal agencies are already expected to work from.