Human-in-the-Loop Controls for High-Risk Agent Actions

A human approving an AI agent's action and a human overseeing an AI agent's behavior are two different things, and most organizations have only built the first one. That gap is the subject of this piece: why a checkpoint is not the same as a governance architecture, and what it takes to close the distance between the two.

Picture an agent that pauses for sign-off before sending an email. Technically, a human is "in the loop." But if the person clicking approve has no visibility into what the agent retrieved, what it reasoned through, or why it chose this action over another, that pause is a formality, not oversight. Agentic AI has changed what's at stake in that formality. Older AI systems produced an output and waited for a person to read it. Agents now plan, decide, and execute multi-step tasks on their own, booking flights, moving money, modifying production infrastructure, so an oversight failure appears immediately, in the real world, rather than sitting in a draft folder waiting for review.

That shift has already produced a measurable governance gap. A survey from Gravitee found that a large majority of enterprises have experienced AI agent security incidents, yet fewer than half of deployed agents are actively monitored. Executive confidence in written AI policy is running far ahead of actual technical coverage, and the incidents on record show what that mismatch costs in practice.

Microsoft's 365 Copilot Chat carried a bug, starting January 21, 2026, that caused the assistant to summarize confidential emails for weeks, bypassing the DLP policies and sensitivity labels the organization had explicitly configured. At Meta, in March 2026, an internal AI agent posted unsolicited advice on a company forum, which triggered a cascade that handed engineers access to systems they weren't authorized to see, with no outside attacker involved at any point. And in the case known as ForcedLeak, an Agentforce agent attempted to send CRM data to an attacker's domain, defeating the only safeguard in place, Salesforce's CSP and Trusted URLs allowlist, through an expired domain that the attacker simply bought back once it lapsed.

None of these three failed because a human was missing somewhere in the org chart. Each one failed because the oversight architecture didn't intercept the specific action that caused the harm. That's the premise the rest of this piece works from: a checkpoint is not a system, and building the system is the actual job.

Oversight tiers map to action risk, not agent capability

The question that decides how an agent's action should be overseen isn't how smart or trustworthy the agent is, but how hard the action is to undo. These three oversight tiers are the organizing principle practitioners now use to structure agent governance, and each one asks something specific of a human.

Human-in-the-loop (HITL) means a person approves the action before it executes. This tier fits high-stakes, irreversible, or regulated actions, financial disbursements, legal agreements, access to sensitive data, where the cost of a mistake outweighs the cost of delay. It is slow by design, and that slowness is the point.

Human-on-the-loop (HOTL) lets the agent act autonomously while a person monitors the stream of activity and can step in when something looks wrong. It trades the certainty of a pre-execution check for the speed of a pause button that's there if needed.

Out-of-the-loop, or full autonomy, removes the human gate at execution time. Nobody needs to sign off before an agent tags a lead as "warm."

| Tier | When the human acts | Speed | Best suited for | |---|---|---|---| | Human-in-the-loop | Before execution | Slowest | High-stakes, irreversible, regulated actions | | Human-on-the-loop | During or after, on exception | Fast, with intervention capacity | Medium-risk, recoverable, high-frequency actions | | Out-of-the-loop | After the fact, if reviewed at all | Fastest | Low-risk, high-volume, reversible actions |

These tiers don't get assigned once, to an agent, as a permanent label. Same agent, three different postures, depending entirely on what it's about to do.

Consider an agent that books a flight and then negotiates a vendor contract within the same task sequence. Treating those two actions identically, because they came from the "same" agent, misses the entire logic of risk-based oversight. The oversight model has to flex within a workflow, not just across workflows.

It's tempting to sidestep all this nuance by requiring approval for everything, every action, every time. Picking the right actions to gate is what keeps the whole tiered system from collapsing into theater.

Which actions warrant human review: a principled trigger framework

Knowing that high-risk actions deserve a human gate doesn't tell an engineering team which actions count as high-risk. That judgment call, made ad hoc, is where a lot of HITL programs quietly fall apart: one team gates spending over some low amount, another gates spending only at a far higher one, and nobody can say why either threshold is right.

Singapore's IMDA Model AI Governance Framework offers a steadier starting point, built around four trigger categories rather than one number. User-defined boundaries cover the thresholds an organization sets for itself: a dollar amount above which a payment needs a human's eyes, a data classification level that always triggers review, a counterparty type that's never approved automatically.

What ties these together is the consequence logic from the tier framework. Each of these categories maps directly onto the blast-radius calculation that decides tier assignment.

Tier assignment doesn't have to be fixed per action type forever, either. Published threshold benchmarks should be treated as a starting hypothesis to test against an organization's own data, not as an industry standard to adopt wholesale, because large language models have been shown to be systematically overconfident when asked to self-report their own probability of being right.

Running the Microsoft, Meta, and ForcedLeak incidents back through this framework makes the gap clear. Copilot Chat summarizing confidential email is exactly the "accessing or exporting sensitive data" trigger, missed because the DLP policy sat outside the agent's actual execution path. A good trigger taxonomy on paper means nothing if nothing in the system actually enforces it at the moment the action fires. That's the handoff to the next question: how does a trigger framework turn into something a running system actually respects?

Oversight structure across the three architectural tiers in practice

A written policy that says "gate all production deployments" does nothing on its own.

That boundary matters because reasoning and action aren't the same event, and only one of them causes damage. An agent's internal chain of thought can wander, misjudge, or guess wrong without hurting anyone, right up until that reasoning reaches a tool call that moves money or exports a file. The gate belongs at that tool call, not back at the chat interface where the reasoning first took shape, because the interception point is the action itself, the tool_call, not the conversation that led up to it.

Anthropic's Agent SDK builds this directly into its architecture through a set of hook events: PreToolUse, PostToolUse, PostToolUseFailure, Stop, SubagentStart, SubagentStop, PreCompact, PermissionRequest, and Notification. The defer decision is the pattern most engineering teams will actually encounter when they build this themselves. When a hook returns defer, the session doesn't terminate. It suspends. The pending decision gets delivered out-of-band, through Slack, a webhook, or a PagerDuty alert, and the session picks back up exactly where it left off once a human approves or denies the action. Nothing about the agent's state gets lost in the pause.

Dify shipped a comparable idea as a native workflow primitive in February 2026, with its Human Input node. Celery workers and Redis Pub/Sub handle the suspend-and-resume mechanics underneath, showing this is running infrastructure, not a conceptual framework someone sketched on a whiteboard. It's running infrastructure.

None of this works, though, if the gate is just a UI convention that a determined action can route around. That's where identity governance earns its place in the stack: binding an agent's actions to identity policies means HITL checkpoints get enforced through actual authentication, authorization, and audit controls, not through a button that a differently-configured request can simply skip. The ForcedLeak incident is the clearest illustration of what happens without that binding. The one control standing between the agent and the attacker's domain was an allowlist, and the allowlist held only as long as the domain's ownership did. Once the domain expired and a buyer picked it up, the gate was still technically "there," but it no longer gated anything.

Most production agent systems aren't a single agent making a single decision; they're chains of subagents, each one potentially operating at a different tier, a research subagent moving freely while a disbursement subagent waits on approval. Keeping that structure coherent, so that oversight discipline holds all the way down the chain rather than just at the top-level agent, is an organizational problem as much as a technical one. Which is exactly the kind of problem that architecture alone can't solve.

Why the rubber-stamp effect turns approvals into liability

A checkpoint that exists but gets approved automatically, out of habit, is more dangerous than no checkpoint. A gate that's routinely rubber-stamped creates something worse: a paper trail showing a human reviewed and signed off on an action that was never actually evaluated by anyone.

This isn't a story about careless employees. Anthropic's own study of millions of Claude Code interactions found that auto-approval rises with experience, not falls. People don't get more careful as they get more comfortable with a tool. They get less careful, because the tool has, by their own experience, earned their trust, and that trust generalizes past the point where it's still warranted.

The International AI Safety Report 2026 gives this pattern a name: automation bias, the tendency for humans in an oversight role to place more trust in an AI system than the system's actual track record justifies. Telling a reviewer to "be more careful" doesn't appear to work as a fix.

There's a useful parallel in aviation's crew training discipline, which treats a pilot's vigilance not as a trait someone either has or doesn't, but as a skill that degrades without deliberate practice and structured habits that counteract fatigue. Oversight in an agent workflow works the same way. A reviewer doesn't stay sharp by virtue of being assigned the reviewer role. Sharpness has to be built into the job, through the volume of the queue, the quality of the information in front of them, and the training they've had, or it erodes.

The clearest early warning sign of that erosion is the false-positive escalation rate: how often an item reaches the human queue and simply gets approved without any change. FutureAGI's operational guidance treats that number as a primary dashboard metric for exactly this reason, not a footnote.

Gating too many actions is itself a documented mistake, and it's the direct cause of the rubber-stamp effect, not a separate problem from it. And in a legal sense, that recorded approval is worse than silence: it is documentary evidence that a human looked at the action and authorized it, when in practice no meaningful evaluation happened.

Designing against rubber-stamping: queue discipline, reviewer training, and graduated autonomy

Treating human review as a skill that needs active maintenance, rather than a structural fact that holds just because a checkpoint exists, changes what an organization has to build. The architecture from earlier in this piece is necessary. It isn't sufficient on its own.

Queue discipline comes first. Every low-stakes action added to the approval queue dilutes the attention available for the one action that actually needs it.

Context has to travel with the request. A reviewer shown nothing but a final tool call, "send $4,200 to this account," has no way to judge whether the action makes sense; a reviewer shown the input, the retrieved context, the agent's planner reasoning, the tool name, the arguments, and the evaluator's confidence score can make that judgment in seconds. Anthropic's own usage data, which found that users approve 93% of permission prompts they're shown, suggests that an approval queue stripped of that surrounding context defaults to near-automatic assent regardless of what's actually in it. The trace is what gives the human something real to evaluate. Without the trace, approval is a reflex rather than a decision.

Mean-time-to-decision should be tracked as an operational metric alongside the false-positive rate from the previous section. Pairing that metric with periodic, deliberate recalibration, pulling a sample of approved actions and asking whether a careful reviewer would have approved them the same way, keeps the review function honest in a way that a one-time training session can't.

Graduated autonomy offers a structural way out of the all-or-nothing choice between full approval gates and full automation. Rather than asking a human to approve every single action an agent takes forever, define the boundaries, sandboxing, auto-approval classifiers, monetary thresholds, within which the agent earns the freedom to act without a per-action check, and reserve the human gate for the actions that fall outside those boundaries. That's the shift Anthropic made internally once its own data showed how often users were approving prompts without real scrutiny: a redesigned boundary around what needs asking in the first place, rather than more warnings stacked onto the same approval flow.

None of this removes the need for judgment. It makes judgment possible again, by giving the human reviewer a queue small enough to read, context rich enough to understand, and a system around them built to notice when their attention has started to slip.