Open-Source LLM Supply Chain Risks in Enterprise Deployments

Open-source LLMs move through four separate handoffs before they ever run in production: pretrained weights, fine-tuning data, plugins and connectors, and the orchestration code tying it all together. Each handoff is a place where trust changes hands, and most enterprise security tooling was never built to see any of them. Here's the position worth stating up front: the layers everyone treats as an afterthought (the hub, the loading library, the fine-tune step) are exactly where the real damage happens, and the layers security teams pour money into (network perimeter, endpoint detection) don't reach any of it.

The enterprise LLM market sat at $6.7 billion in 2024, hit $8.8 billion in 2025, and is on track for $71.1 billion by 2034, a 26.1% compound annual growth rate. Enterprise AI spending overall jumped from $11.5 billion in 2024 to $37 billion in 2025, a 3.2x jump in a single year. Research tracked by BarChart/LLM.co puts open-source AI tool adoption above half of enterprises, drawn in by lower licensing costs and the freedom to run models on their own hardware. Closed-source models still handle roughly 87% of production workloads, but open-source has taken over development, fine-tuning, and internal data science work: the exact corners of the stack where a compromise is hardest to catch. Gartner expects more than 70% of organizations to have AI models built into their workflows by the end of 2025.

Here's the tension people keep skating past: open-source's biggest selling points (cost, control, the freedom to customize a model to the exact task) are the same properties that make the supply chain hard to police. You don't get one without the other. The rest of this piece walks the chain link by link and asks, at each one, who's actually watching it. Spoiler: mostly nobody, and the layers where nobody's watching are the ones where the real damage happens.

How the open-source LLM supply chain is actually structured, and why each handoff matters

OWASP's LLM03:2025 entry lays the supply chain out plainly: pretrained model weights, fine-tuning adapters like LoRA and PEFT, training and fine-tuning datasets, plugins, connectors, orchestration middleware, open-source Python packages, and the CI/CD tooling that ships all of it into production.

Here's what makes this different from a normal software supply chain, and it's the point most security teams miss first. Traditional dependency risk is about code: a library with a bug, a package with a backdoor. The ML supply chain carries components that encode behavior, not just instructions. A poisoned dataset or a tampered weight file changes what a model does without a single line of code changing anywhere a standard scanner would look.

Each handoff is its own trust boundary. Who published the base weights? Who curated the fine-tuning data? Who maintains the library that loads the model, and who controls the connector reaching into enterprise systems? Standard software composition analysis tools have no answer to any of these questions, because they were built to answer a different one entirely.

OWASP makes a point worth sitting with: companies that only consume LLM APIs through a middleware layer or a proxy gateway are still inside this supply chain. LLM03 isn't a problem reserved for teams training their own models from scratch. That's easy to miss if "supply chain risk" sounds like something that only applies to the people building the base model.

A single poisoned upstream component cascades silently through everything downstream of it. A malicious weight file becomes a production model. A backdoored fine-tune ships inside a customer-facing product. A compromised dependency exfiltrates credentials the moment inference runs. The sections ahead move through this chain in order: the hubs models get pulled from, the naming tricks that hijack pipelines, the loading library itself, the CI/CD layer around it, and the fine-tuning step where behavior gets rewritten on purpose.

The model hub as an untrusted package registry: what Hugging Face's threat landscape actually looks like

Treat Hugging Face like what it functionally is: a package registry with no gatekeeper. Anyone can publish a model under any name. Provenance is whatever the uploader says it is, and platform scanning, however well-intentioned, has documented gaps.

The MalHug detection project watched more than 705,000 models and 176,000 datasets on a mirrored Hugging Face instance over three months. It found 91 malicious models and 9 malicious dataset loading scripts, with payloads ranging from reverse shells to browser credential theft to basic system reconnaissance.

ReversingLabs found two malicious models that slipped past the platform's own safety scanning entirely. Both used Pickle file serialization, a format that runs arbitrary code the moment it loads. Those two repositories sat undetected for more than eight months. This is the core risk at this layer, and it's worth saying without softening: there is no safe way to load an untrusted Pickle file. Not a safer way. None.

Naming is its own battlefield. A study covering 1,020,755 models, 219,812 datasets, and 127,011 organizations on the platform found 1,574 potentially malicious squatting models, with 10.4% showing suspicious or outright harmful traits. Dataset squatting turned up 625 cases, 42.2% of which looked like intentional impersonation rather than accident.

Then there's the social layer, probably the most human part of the whole problem. A fake OpenAI model calling itself a "Privacy Filter" made it into Hugging Face's trending section and racked up hundreds of thousands of downloads before anyone flagged it. Nobody got fooled by anything clever here. Developers just trusted a familiar name over checking where it actually came from, which is exactly the behavior trending lists and recognizable org names are built to encourage.

Add it up and the picture isn't a handful of unlucky uploads. It's systematic, patient, and dressed up in the same discovery signals developers rely on to find models worth using in the first place. And here's the part worth sitting with: none of these three failure modes (malicious payloads, name squatting, trending-list impersonation) needs a sophisticated attacker. They need a platform that treats "anyone can upload" as a feature rather than a risk to manage.

Orphaned model namespaces and the pipeline hijack they enable

Walk through this one slowly, because it isn't intuitive at first. A model gets deleted or transferred on a cloud provider's catalog. Its old path, the exact string a pipeline used to reference it, goes up for grabs. Anyone can re-register that same name. If a pipeline resolves models by name instead of by a cryptographic hash, it pulls down whatever now sits at that address, no questions asked.

Palo Alto's Unit 42 documented exactly this risk: an abandoned namespace can be re-registered, recreating the original path, and delivering arbitrary code to any pipeline still resolving models by name instead of by digest.

This isn't a one-off bug in one product. It's systemic wherever automation trusts a mutable name over an immutable identifier. After Unit 42 flagged the issue to Google in February 2025, Google rolled out daily scans for orphaned models. That fixes one hub operator. It says nothing about every other hub, and nothing about the MLOps tooling most enterprises run internally.

Separate this clearly from typosquatting, because the mechanics run in opposite directions. Typosquatting hands you a fake name close enough to fool you. Namespace reuse hands you the real name, the one you already trusted and already wrote into your pipeline config, after the legitimate model behind it is gone. Standard name-matching checks miss this entirely, since the name matches perfectly. Every unpinned model reference in a production pipeline is a small bet that nobody will ever re-register that path, and some of those bets have already lost. If your pipeline config still has a bare model name in it instead of a hash, that bet is running right now, whether anyone remembers placing it or not.

When the vulnerability is in the loading library itself: CVE-2026-4372 and the trust_remote_code bypass

Now the layer underneath all of this: the library that actually loads the model. A vulnerability found in Hugging Face's Transformers library let attackers run remote code through a malicious model configuration file, and it did so by walking straight past trust_remote_code=False, the exact setting developers flip on specifically to stop this.

The mechanism asks for almost nothing. The mechanism works through a malicious value injected into the model's configuration file. Any user who then calls the standard from_pretrained() function, the everyday way models get loaded, triggers arbitrary Python execution. No unusual click, no special permission, no odd behavior asked of the victim. Just the normal workflow, running normally.

The vulnerable code path remained exploitable for an extended window during which any enterprise pulling affected models could get compromised without anyone noticing. The nature of the vulnerability meant exposure extended beyond a sandbox compromise, representing a potential foothold into enterprise infrastructure.

Here's the detail that raises the stakes further: the vulnerability affected the kinds of Transformers installations most common in production inference and fine-tuning clusters, the highest-value targets on the network.

The lesson underneath the CVE outgrows the CVE itself, and it's the one worth remembering after the patch notes are forgotten: a safety setting living inside someone else's library isn't a control you own, it's a promise you're trusting. trust_remote_code=False was supposed to be the wall. It turned out to have a door in it nobody checked. This isn't an isolated case either: Unit 42 disclosed two other high-severity vulnerabilities, one in NVIDIA's NeMo (CVE-2025-23304, patched in NeMo 2.3.2) and one in Salesforce's tooling (CVE-2026-22584, fixed July 31, 2025). The LLM tooling ecosystem has more than one active front open right now.

How CI/CD pipelines and Python dependencies extend the attack surface beyond the model itself

CI/CD is probably the single most exploitable node in the whole LLM DevOps stack, and the reason is structural. It sits at the intersection of model registries, dependency managers, cloud credentials, and deployment targets, all wired into one automated flow. Compromise that one node, and everything downstream of it is reachable.

March 2026 gave a real example. Attackers broke into LiteLLM, a widely used LLM proxy gateway, through a single poisoned package on PyPI. Sensitive credentials were exfiltrated before anyone caught the breach.

The structural problem underneath the LiteLLM incident is transitive dependencies. Installing one package pulls in its entire dependency tree, sight unseen. One bad package, malicious on purpose or just quietly vulnerable, propagates across every downstream project, and there can be thousands of them.

Attackers have also started going after the update mechanism itself instead of hunting for code bugs. Attackers have also targeted maintainer accounts and release infrastructure directly, self-replicating through open-source registries by compromising the pipeline that ships trust, not the code that trust ships through.

One useful lesson out of LiteLLM's containment: limiting what a compromised environment can reach outbound is a meaningful second layer of defense even after a malicious package runs. Prevention fails sometimes. Layered defense is what catches it when it does, and too few AI teams have built that second layer.

There's an organizational gap sitting underneath all the technical detail, and it might be the bigger problem. Plenty of AI teams manage Python packages differently than they manage application code: requirements pinned by name and version instead of by hash, dependency review skipped in ML pull requests, model-loading code treated as somehow exempt from the scrutiny application code gets as a matter of course. Security teams tend to own application CI/CD while ML engineers own model pipelines. The handoff between those two groups is often exactly where nobody's controls actually apply. Ask which team at a given company owns the security review for a Hugging Face download landing in a training job, and don't be surprised if the honest answer is nobody.

Fine-tuning as an attack surface: backdoors, guardrail removal, and the regulatory gap they exploit

Weight tampering (bit flips, file hijacking, module insertion) lets an attacker force a model into unintended behavior without touching a single line of the application code around it. Integrity checks built at the application layer never see this, because there's nothing at that layer to check.

Fine-tuning backdoors work on the same basic idea, just through training instead of direct file manipulation. Train a model on a dataset seeded with a trigger phrase, and it acts normal on every ordinary input while producing an attacker-chosen output the moment that trigger shows up. Nothing in an inference log flags this. The trigger stays invisible until someone actually types it.

How fragile is safety alignment against this? Research showed that fine-tuning on as few as 10 samples could jailbreak GPT-3.5. That's not a typo: ten examples, and a model's safety training comes apart. A model card's language about safety testing tells a reader almost nothing about how that safety holds up after someone runs a small, targeted fine-tune on top of it.

As of July 2025, more than 3,000 fine-tunes of Llama-2 7B and over 1,000 fine-tunes of Qwen 2.5 7B sit on Hugging Face for anyone to download, and most carry little to no documentation about what data trained them or whether the original safety properties survived. That's thousands of unverified variants of two base models alone, multiplied across every other open-weight model on the hub.

Regulation hasn't caught up to this gap, and it's worth being blunt about where it fails. The EU AI Act's GPAI Guidelines require disclosure only for a "significant modification," but stripping out safety guardrails takes negligible compute, well under whatever threshold counts as significant. The rule was written for large-scale retraining. The actual danger lives in small, cheap, targeted changes the rule never reaches, so passing the letter of the regulation tells you nothing about whether the guardrails survived the fine-tune.

Multi-turn attacks make the picture worse. Cisco's AI Validation assessment found a an exceptionally high success rate for jailbreaks unfolding across multiple turns of conversation. Models holding up fine against a single hostile prompt often give way once the conversation stretches out, since nothing in most architectures enforces safety constraints across the full arc of a dialogue.

Put plainly: a fine-tuned model pulled from a public hub comes with no reliable record of what was actually done to it. The model card is self-reported. The weights are opaque. Standard behavioral testing doesn't reliably surface a backdoor sitting there waiting for its trigger. And most existing defenses assume harmful behavior has to be learned through fine-tuning in the first place, when pretrained models already carry plenty of harmful knowledge baked in from pretraining, knowledge that techniques like abliteration can expose without any further training at all. If you're only checking for what fine-tuning added, you're missing what pretraining already put there.

Why traditional security tooling has no visibility into most of this

Software composition analysis tools were built to match a package name and version against a CVE database. Fine tool, for that job. It has no schema for a model weight file, no concept of dataset provenance, and nothing resembling a way to flag a fine-tune that quietly stripped out safety alignment.

SAST and DAST tools inspect code paths, which made sense for the job they were originally built for. But the danger in a malicious config.json or a poisoned Pickle file isn't a code path at all. It runs before any application logic even starts, inside the model-loading call itself, a place these tools were never pointed at.

Third-party risk management leans on vendor questionnaires and penetration test reports. Neither one asks whether a vendor's model was fine-tuned on poisoned data, or whether its connector quietly reaches out somewhere it shouldn't.

The Hugging Face numbers make the gap concrete: 91 malicious models found in a three-month scan, missed by the platform's own scanning. If platform-native detection struggles at that scale, enterprise security teams looking in from outside have far less visibility, and most don't have a process for scanning model files at all.

CVE-2026-4372 is the cleanest single picture of the whole problem. Six months of exposure in a library used everywhere, during which the library's own advertised safety control got quietly bypassed. Standard vulnerability management wouldn't have caught any of it until a patch already existed to fix it.

The org chart makes this worse, not better. InfoSec owns application security. ML engineers own the model pipeline. Legal and Privacy own the vendor contracts. The AI supply chain runs straight through all three of those domains and isn't fully owned by any single one of them. Generic governance frameworks (ISO 27001, SOC 2, NIST CSF) give a company a compliance surface to point to, but none of them contain a control built specifically for model weight integrity, fine-tune provenance, or prompt injection arriving through a connector. Passing the audit and closing the actual gap are two different things, and most enterprises are only doing the first one.

What a defensible enterprise posture across each layer actually requires

Start with sourcing, and be strict about it. An approved model registry, where only vetted models run in production, flips the default from "anything downloadable is fair game" to "only what's been checked gets in." Hub downloads should land in a staging environment with scanning first, never straight into a live pipeline.

Pin every model reference to a cryptographic hash or digest, never a mutable name. This one change closes the namespace reuse problem outright, since an attacker can re-register a name but can't forge the hash of the original file.

Prefer SafeTensors over Pickle wherever a model supports it, and treat that as non-negotiable rather than a style preference. SafeTensors was built specifically to avoid the arbitrary code execution risk baked into Pickle's design. Given that Pickle has no safe workaround at all, format choice stops being a preference and becomes an actual control. Anyone still defaulting to Pickle for convenience is making a call they haven't actually thought through, and that's the wrong call every time.

Beyond sourcing, the fine-tuning layer needs its own scrutiny: provenance tracking for training data, behavioral testing that specifically probes for trigger-based backdoors rather than just standard capability benchmarks, and real skepticism toward any fine-tune whose data lineage isn't documented. The CI/CD layer needs dependency pinning by hash, review steps for model-loading code that match what application code already gets, and network-level egress controls that catch a compromised package even after it's already running.

None of this asks anyone to walk away from open-source models. It asks that each handoff in the chain get treated as a decision someone actually made on purpose, instead of a default nobody bothered to question.