Runtime Behavioral Scoring to Distinguish Exfiltration Probes From Benign LLM Queries

Detecting attempts to extract confidential system prompts or data from a deployed LLM requires runtime behavioral scoring of queries, tool-call sequences, outputs, and session history. Input filters alone cannot tell a harmless request from one that turns into an exfiltration probe as it runs. Effective detection combines fast rules, contextual reasoning, and, where inference access permits, token-level verification, with boundary monitoring for vendor-managed services.

Why static and input-layer controls cannot see what runtime scoring sees

Static code analysis and dependency scanning operate before inference. They can examine code and dependencies, but they cannot observe how a model responds to a live user, which tools it calls, or what those tools return. Input-layer filters check the query as it comes in, with prompt classifiers and blocklists judging the query itself. When a request looks ordinary until the model starts acting, the distinguishing evidence does not yet exist at the input checkpoint.

The Marimo intrusion makes that gap concrete. The attacker’s agent inferred non-standard database table names and used outputs from earlier commands to shape later actions. Its path depended on what it discovered. A static rule could not anticipate that exact sequence because even the agent did not know its next discovery in advance. Runtime scoring handles this uncertainty by checking behavior as it unfolds, rather than relying only on a prediction made before execution.

What the behavioral signals of an exfiltration probe actually look like at runtime

In the Marimo case, the evidence was a cluster of behaviors, including machine-readable output formatting, rapid database queries, and commands chained from earlier outputs. Delimiter injection, bounded output, and HEREDOC batching made results suitable for subsequent machine processing. These conventions are not proof by themselves. Their significance grows when an agent repeatedly uses them while querying tables whose names never appeared in prior session context.

The same intrusion distributed API calls across eleven Cloudflare Workers egress IPs to defeat source-IP detection. A request-by-request view fragments that activity into apparently separate sources, while a session view reveals the coordinated sequence. Timing and history change the interpretation. A query that looks benign alone becomes more suspicious when it follows credential harvesting or environment enumeration, because its role depends on what the session has already accomplished.

Steganographic exfiltration presents a different kind of evidence. Instead of exposing suspicious command sequences, an attacker encodes hidden information in otherwise ordinary model outputs. Rinberg et al.’s inference-verification framework formalizes how this encoding must deviate from the expected generative process, creating a token-distributional signal. That signal requires a different vantage point. Detecting it means supplementing inspection of the text for visible secrets or suspicious formatting with scoring tokens against what the claimed model could honestly have produced.

How runtime behavioral scoring is structured to capture these signals

A layered scorer separates quick, predictable decisions from judgments that need context. Its deterministic tier handles known formatting patterns, rate anomalies, and source-IP distribution, while semantic classifiers and an LLM reasoning tier assess less obvious combinations. Context links events across tool calls, hosts, and detection cycles that would be unclear on their own. Not every event needs the reasoning tier.

BlueSTAR demonstrates this architecture in enterprise IT/OT cyber ranges. Its deterministic tier provides fast remediation for known-pattern threats, while its reasoning tier correlates evidence across cycles to address attacks involving credential theft and concurrent attackers. Crucially, the system first transforms high-volume telemetry into compact indicators of compromise, because raw logs arrive faster than an LLM can usefully process them and individual events are often ambiguous.

Rinberg et al. address a separate problem at the inference layer. Their Token-DiFR and Token-IPT-DiFR estimators assess whether generated tokens are consistent with honest inference by the claimed model. This supports information-theoretic bounds on how much hidden information an attacker can transmit. The attacker faces a speed-detectability trade-off. The framework deals with steganographic model-weight exfiltration from an inference server, so its findings do not prove that checking ordinary API traffic can detect every concealed payload.

Alghamdi et al.’s University of Queensland framework addresses the abstraction between logs and a classification decision. It represents activity as chronological timelines, uses retrieval-augmented generation to supply each user’s historical context, and generates interpretable, threat-specific risk indicators. Those indicators are then classified together across sequences of events over time. Indicator quality drives the resulting detection performance. This approach gives the final scorer structured evidence instead of asking it to infer an entire attack directly from an undifferentiated stream of logs.

What real intrusions reveal about where behavioral scoring must be positioned

EchoLeak, CVE-2025-32711 in Microsoft 365 Copilot, shows why inspecting the user’s direct request is not enough. One crafted email made zero-click, remote, unauthenticated data exfiltration possible through prompt injection. Output behavior and the tool-call sequence were therefore the relevant scoring surfaces, rather than a user prompt that explicitly requested theft.

Claude Code’s CVE-2025-55284 illustrates the output-pattern requirement through DNS-based exfiltration triggered by an injection in a repository file. Data appeared in DNS queries as encoded subdomains, including a secret embedded in a single hostname. CVE-2026-54316 used a slower covert channel. The latter pattern calls for cumulative session scoring, because thresholds aimed at bulk data movement can miss a sequence of tiny transfers.

Marimo exposes the same need through adaptive behavior. Its initial intrusion session differed from the subsequent agent-driven exfiltration phase through machine-speed queries, output-chained commands, and distributed egress. Those changes were visible in the sequence. Separately, the September 2025 Postmark MCP supply-chain rug pull placed the signal in output traffic when the impersonation package postmark-mcp added unexpected blind-carbon-copy recipients in version 1.0.16 after fifteen clean versions. Version history built trust, but watching outgoing behavior was the way to detect when the package broke that trust.

How MCP server and agentic connector architectures expand the behavioral scoring surface

A single MCP session can connect an agent to dozens of servers spanning file systems, GitHub, Slack, Stripe, and databases. This broadens more than access to data. Through tool poisoning or cross-server tool shadowing, a malicious server can inject tool descriptions that change how the agent understands trusted tools alongside it. Consequently, scoring one connector in isolation can miss the relationship between an instruction received through one server and an action taken through another. The relevant unit becomes the full tool-call graph.

Developers still need controls that cover what the agent can do through those connections. Marimo’s distributed-egress pattern reinforces the distinction between network visibility and session visibility, because changing source IPs can defeat a source-based detector without changing the underlying sequence of tool calls and outputs.

What governance frameworks require and where they leave runtime scoring to practitioners

The NIST AI RMF provides a governance framework without prescribing technical runtime enforcement controls. NIST’s Center for AI Standards and Innovation launched the AI Agent Standards Initiative in February 2026 to address agent-specific risks, but that does not supply an implementation for scoring a live tool chain. Frameworks and detectors serve different purposes. A risk taxonomy can establish what an organization must consider without deciding which signals its deployment can observe or how those signals should be combined.

That separation leaves practical decisions with the organization and its tooling providers. How much session history must a scorer retain, and which evidence justifies escalation? Those questions depend on the runtime architecture. In financial services, DORA adds a third-party oversight context that makes ongoing monitoring especially relevant, but it does not replace the technical design of a behavioral scorer.

How third-party vendor AI deployments create runtime scoring blind spots that internal controls cannot fill

A vendor-managed LLM changes what the deploying organization can see. Without direct access to the inference pipeline, it cannot simply install token-level verification inside the service. Its scoring surface is the observable boundary, including queries, tool-call invocations, outputs, and the session context available around them. This is a real limit on detection. Boundary monitoring can correlate visible behavior, but it should not be presented as equivalent to inspecting the vendor’s inference internals.

Vendor changes also complicate the meaning of a baseline. A vendor can change default settings, subprocessors, retention policies, or model versions, and an unannounced model change can shift the behavior against which scoring was calibrated. The Postmark case illustrates the supply-chain version of this problem, where a trusted history preceded a malicious release. Continuous monitoring addresses the period after assessment, when the previously reviewed service or dependency may no longer behave as expected.

The European Supervisory Authorities published DORA’s Article 31 Critical Third-Party Providers list on November 18, 2025, with 19 providers on it. Organizations that use AI services from those providers must keep monitoring them, which point-in-time assessments cannot do. The operational need extends beyond a questionnaire. Dedicated AI risk intelligence platforms that assess and continuously monitor vendor AI assets can address this gap by surfacing emerging threats across the vendor ecosystem. Their role complements internal runtime controls rather than claiming access to an inference layer the customer cannot inspect.

An inference verifier addresses token-level behavior where the required access exists, while a session scorer correlates queries, actions, and outputs. A vendor-risk platform addresses ongoing exposure across externally managed AI assets. Evaluating them against the observable signal surface keeps a layered detection plan from becoming a collection of tools that all watch the same narrow entrance.

What implementing runtime behavioral scoring requires in practice

The practical sequence starts with signal grounding. Raw inference and security traffic must be turned into compact indicators before it reaches the reasoning tier so that logs do not overwhelm that tier. BlueSTAR demonstrates why this preprocessing is necessary on live infrastructure. Session memory comes next, retaining tool calls, prior outputs, and actions across detection cycles so the scorer can distinguish a new probe from a persistent one. Together, these layers give the system evidence it can process and enough history to interpret it.

Layered detection then combines deterministic rules for known output patterns and rate anomalies with semantic classifiers and cross-cycle reasoning. Threshold calibration must address both false positives and attack coverage. Alghamdi et al.’s findings support putting work into better generated risk indicators rather than assuming a stronger underlying model will make up for poorly prepared evidence. Covert channels need separate treatment. Token-level verification belongs at the inference layer for steganographic threats, while slow external channels such as download counters need cumulative output-pattern analysis because thresholds set only for large transfers can miss them.

For vendor AI, implementation shifts to the boundary the organization can observe. Cross-vendor telemetry and continuous baseline monitoring must track changes in model behavior or default settings while respecting the limits on what they can see of internal inference. The final step is governance mapping. Scoring outputs should map to OWASP LLM Top 10 2026 categories LLM01, LLM02, and LLM03, and MITRE ATLAS identifiers AML.T0054, AML.T0051, and TA0015. That translation makes the evidence interpretable to TPRM, information security, Legal, and compliance teams, so an engineering detection can support an organizational decision.