Runtime Behavioral Baselines for Detecting Adversarial Query Campaigns Against Vendor LLM APIs

Detecting adversarial query campaigns against vendor LLM APIs requires continuous runtime behavioral baselines. For system prompt confidentiality and data exfiltration risk detection, the main layer compares query-stream patterns over time and across sessions, while other monitors check how the model behaves where they have access. These methods reveal signals that are hidden when requests are viewed alone, but no method on its own can guarantee that confidential instructions will not leak.

Why per-query detectors cannot see adversarial query campaigns

Mahalanobis distance, energy-based out-of-distribution scoring, and text anomaly detectors can score each request against an expected pattern. But when an extraction query looks like a legitimate one, the score has no campaign context to distinguish them. The missing information sits outside the request. Tightening the threshold does not give the detector access to that information.

The attack draws its power from the relationship among requests, while the detector evaluates each request separately. Rate limits and stricter per-account thresholds address volume within their own boundaries, but a well-resourced actor can spread activity across accounts. Each account can remain within normal bounds while the combined stream serves an adversarial goal.

What active campaigns actually look like at the query-stream level

Google's Threat Intelligence Group said in a February 2026 report that it detected and disrupted attempts at model extraction against Gemini throughout 2025. One campaign used a large number of prompts to replicate reasoning across non-English languages. The campaign’s purpose emerged from its collective activity. When that many prompts are spread across time and accounts, looking more closely at each prompt would not be enough to flag the volume as suspicious.

Operation Bizarre Bazaar illustrates a different kind of shared signal. A single JA4H signature appeared in the vast majority of attacks across dozens of source IPs in many countries, pointing to shared automation tooling. The locations differed, but the fingerprint persisted. A detector confined to one session could observe that session’s signature without seeing its recurrence elsewhere. The useful evidence was the relationship across sessions and infrastructure, which isolated scoring could not establish.

LLMjacking adds unauthorized access to this picture. The theft and unauthorized use of AI API credentials has developed from opportunistic cost-shifting into professionalized criminal infrastructure. The initial campaign targeted multiple providers, including AWS Bedrock, Azure OpenAI, Anthropic, and Google Vertex AI. Together, these campaigns show why the relevant observation boundary cannot automatically be equated with a prompt, an account, or a single vendor endpoint.

Why the same gap applies to backdoors, jailbreaks, and prompt injections — not just extraction

Extraction is not the only threat whose intent can be difficult to infer from an input. Training-time backdoors remain dormant until a trigger appears, so clean-data validation may never exercise the behavior that makes them dangerous. Jailbreaks use adversarial framing that can resemble ordinary requests. Prompt injections instead put instructions that override the task inside content the model has been asked to process. In each case, the input’s apparent normality can conceal its effect on the model.

Layerwise Convergence Fingerprinting’s authors describe the shared difficulty directly. They write, “it is challenging to tell from the prompt alone whether it will cause misbehavior, and ordinary clean-data validation during evaluation fails to catch it”. This distinction is important for detecting attempts to extract confidential system prompts. Recognizing suspicious wording is not the same as recognizing that the model is being diverted from its instructions. Runtime behavior provides evidence that checking prompts alone cannot give, though this does not prove that every behavioral detector can catch a system-prompt leak.

Existing runtime defenses often address one threat at a time and assume a clean reference model, known triggers, or editable weights. Enterprise consumers of opaque vendor APIs rarely have those conditions. Misbehavior can also be introduced at the provider level through fine-tuning, quantization, or misconfigured system components. The broader requirement is therefore behavioral monitoring from several vantage points.

Three technical approaches to building runtime behavioral baselines

Traffic-window distribution testing takes the API-side, black-box vantage point. The EMNLP 2026 formulation embeds incoming queries into a semantic space and uses Maximum Mean Discrepancy, or MMD, to compare their aggregate distribution with historical benign traffic. The defender builds a benign reference pool, compares batches of benign queries to build a null distribution, and flags incoming windows with scores above a high-percentile threshold. The window is the key design choice. It preserves the aggregate pattern that disappears when each query is judged alone.

The traffic-window evaluation covered fourteen attacker-normal pairs and reported near-zero benign false positives, perfect detection on pure-attacker traffic, and strong performance when attacker traffic was diluted. Those results support the window as a useful detection unit. Access defines where this tool fits. It is best suited to an API operator observing incoming traffic, while an informed adversarial provider could detect and reroute a special audit-query distribution. An enterprise auditing its supplier therefore faces a different problem from an operator detecting extraction.

Nay Myat Min, Long H. Pham, and Jun Sun at Singapore Management University introduced Layerwise Convergence Fingerprinting, or LCF, as a server-side monitor. It treats the trajectory of hidden states between layers as a health signal, computing a diagonal Mahalanobis distance on each inter-layer difference, aggregating through Ledoit–Wolf shrinkage, and calibrating through leave-one-out comparisons on clean examples. It needs no reference model, trigger knowledge, or retraining. Unlike input-only scoring, its measurements concern what happens inside the network as the model processes the request.

LCF found anomalies at different depths for different threats. Jailbreaks appeared at early layers, prompt injections at middle layers, and backdoors at architecture-specific depths in the later half of the network. A single all-layer aggregation caught all three across four tested architectures without tuning for each threat, and the measured inference overhead was very low. That reach has an access condition. A consumer with only a vendor API cannot assume access to hidden states, and calibration on 200 clean examples raises concerns about how long the baseline remains representative in production.

Researchers at USC and UC Berkeley published a black-box auditing method at ICLR 2026 to check the behavior of models served by providers. It tests whether a black-box API behaves like a faithful local reference, using a single API call per prompt and natural, user-like queries. Its enterprise use is repeated auditing for degraded or substituted models. Its reference requirement is substantial. The auditor needs full access to a faithful implementation, including decoding parameters and logit outputs, so the method focuses on open-weight models rather than offering a universal test for proprietary APIs.

These approaches answer different questions about the deployment. Traffic testing asks whether incoming requests collectively depart from normal use, LCF examines internal processing, and RUT tests whether the served model matches its reference. They are complementary. For system prompt confidentiality and exfiltration risk, the distinction prevents a category error in which model-integrity auditing is presented as direct proof that malicious queries or leaked outputs will be caught. The choice depends on what each method can see, what reference it needs, and which behavior it measures.

Why distributed, multi-session campaigns defeat even stream-level baselines applied session by session

Distributed attacks can span independent agents, teams, accounts, and runtimes, leaving each local monitor with a sparse fragment of the activity. Each fragment may appear unremarkable against its local baseline. The same structural problem then returns at a larger scale. Moving from prompts to sessions is progress, but it does not establish whether separate sessions belong to one campaign.

Park et al. (2026) introduced Asynchronous Attribution Fingerprint Vectors, or A²FV, to address this linking problem. The lightweight proxy-side protocol uses tool-use structure, inter-call timing, and prompt surface style to link sessions that belong to the same campaign. Its central finding is that campaign identity leaves recoverable residue in proxy-observable logs even when sessions share no runtime state, arrive asynchronously, and mix with benign traffic. Score-only adaptations of per-session detectors and chunked LLM judges remained near chance at campaign linking.

That research addresses the difficulty of linking activity across sessions visible in Bizarre Bazaar. The campaign used 62 source IPs in countries across multiple continents, and consistent JA4H fingerprints linked activity that would look like separate pieces if each session were viewed on its own. A related GreyNoise SSRF campaign targeting LLMs involved VPS-based infrastructure, but that detail belongs to the separate campaign. The distinction keeps the evidence precise. A²FV shows why recoverable cross-session residue is a detection opportunity that stronger local scoring cannot replace.

Where current baseline detection methods are still vulnerable

An informed adversary can target the baseline itself. MMD-based extraction detection becomes vulnerable when attackers know the mechanism and construct queries that resemble the benign distribution, with detection weakening at low attacker fractions. RUT uses natural, user-like queries to avoid recognizable audit patterns, but it assumes the auditor controls the query distribution. That condition may not hold in every enterprise deployment.

A²FV has a related limit under adaptive attack. Its work primarily addresses static evasion, while an adversary with direct access to its scoring system could optimize activity to suppress the observable residue used for linking. LLM-based oversight has another weakness, systematic benign bias. Detector models are more likely to label adversarial behavior as benign when the models are smaller or their observation windows are shorter. Consequently, placing an LLM judge over a short fragment does not necessarily repair the context that earlier detection layers missed.

Normal traffic can undermine a baseline too. Product evolution and changing user populations alter the behavior a production system should regard as ordinary, making LCF’s fixed clean calibration potentially brittle over time. Traffic-window testing faces the same reference problem. Without continuous updates to the benign pool, legitimate changes can begin to look anomalous. A baseline fixed at onboarding is thus a poor fit because it cannot keep up with changes in normal use or attacker adaptation, while continuous recalibration can keep up with normal use but leaves the risk of attacker adaptation.

Why onboarding-time assessment alone cannot substitute for continuous baseline monitoring

Vendor LLM APIs are not static artifacts. RUT’s authors warn of an ongoing risk that providers may serve quantized or fine-tuned variants to cut inference costs or change behavior. A check at onboarding establishes only what was served during that check. It cannot establish behavioral integrity after a later substitution. Repeated auditing follows from the threat model.

LCF’s authors place the runtime decision at prefill, before any token is emitted. The monitor evaluates behavior when a request is processed, while onboarding evaluates a vendor at a particular point in the relationship. Neither performs the other’s job. Monitoring cadence must therefore match the activity being assessed, with inference-time checks for runtime behavior and continuing observation for campaigns that unfold across requests and sessions.

For TPRM, InfoSec, and Legal teams, the implication is that a point-in-time assessment cannot discharge vendor AI risk as though it were a settled property documented in a SOC 2 report. Models, infrastructure, user populations, and adversarial activity can change after approval. Platforms combining vendor risk assessment with real-time AI threat detection are structurally better positioned to observe those changes than governance frameworks built on periodic review alone. Their advantage depends on actual visibility across the vendor ecosystem. Continuous monitoring is necessary because the evidence needed to detect the threat is produced during use, after onboarding has ended.