Third-Party AI Model Risk Assessment

Start with security, because that's where most conversations begin. Prompt injection gets the most airtime, and deservedly so. An attacker embeds hidden instructions inside user inputs, overriding the model's guardrails, redirecting its behavior, or coaxing it into sharing data it shouldn't. This is especially acute in LLM-based vendor products where user-supplied text flows directly into model context with minimal sanitization. Related to this are adversarial inputs: carefully crafted data that deceives the model into incorrect outputs, sometimes in ways that are deliberately hard to distinguish from normal behavior.

Then there's the supply chain attack surface, which most organizations haven't fully absorbed yet. The vendor's codebase, the pre-trained base model they're building on, the upstream APIs they depend on, all of these are entry points that exist before the model ever touches your environment.

But what if the more dangerous exposures aren't the ones making headlines? Here's what I've noticed in practice: security vulnerabilities, as serious as they are, aren't where most organizations actually bleed. The day-to-day exposure lives in model-quality failure modes.

Hallucinations are the obvious one. Vendor models produce confident, fluent, factually wrong outputs. When those outputs are embedded in customer-facing products or decision-support workflows, the deploying organization owns the consequences, not the vendor. Model drift is subtler and, in some ways, more insidious. Accuracy degrades as the world changes and training data grows stale, and there is often no visible signal to the deployer. The outputs don't fail loudly; they just become gradually less reliable, which is its own kind of danger.

Bias in training data deserves separate attention. Amazon's internal recruiting tool, eventually scrapped after it was found to systematically penalize résumés containing the word "women's," is the case study that keeps coming up in these conversations, and for good reason. The discrimination wasn't intentional; it was a structural artifact of what the model learned from historical hiring patterns. The tool was operating at scale before anyone caught it. That's the thing about bias: it's often invisible until it isn't. But why exactly does this happen? Because bias doesn't announce itself — it compounds quietly inside systems that appear to be functioning normally.

Transparency failure compounds all of this. When vendors treat model internals as a black box, it becomes difficult to detect bias, explain outcomes to affected individuals, or satisfy regulators who increasingly require explanations. FINRA and financial regulators have specifically flagged explainability as an adoption obstacle.

There are also data flow risks that simply don't exist with conventional SaaS. Many AI vendors ingest, log, or use customer inputs for retraining. If that isn't clearly addressed in the contract and actively monitored, you don't actually know what data is leaving your environment or how it's being used downstream.

And shadow AI. Employees adopting vendor AI tools outside sanctioned procurement channels. IBM's 2025 Cost of a Data Breach Report found shadow AI was a factor in 20% of breaches and added roughly $670,000 to average breach costs.

What connects all of these is that most are invisible to perimeter security tools and invisible to standard vendor questionnaires. They require assessment mechanisms built specifically for AI, which is the argument for everything that follows.

How Exposed Organizations Actually Are Right Now

The breach trajectory is worth sitting with. Verizon's 2025 DBIR found that third-party-involved breaches had risen to 30% of all incidents, up from roughly 15% the prior year. A doubling in one reporting cycle.

AI-specific exposure is already inside that number. IBM's 2025 Cost of a Data Breach Report found that 13% of organizations had experienced breaches involving AI models or applications, and 97% of those organizations lacked proper AI access controls. I'll let that sit for a moment. Nearly every organization suffering an AI-specific breach had failed to implement basic access controls for the AI systems involved. Not advanced monitoring. Not behavioral analysis. Basic access controls. That raises an important question: if the failures are this fundamental, what does that tell us about how seriously organizations are actually treating AI vendor risk?

The financial stakes attach quickly. Secureframe's research put average costs for supply chain and third-party vendor compromise at $4.91 million per incident in 2025. Deloitte's 2024-2025 TPRM research found that nearly half of respondents believed potential damages from a major third-party incident exceed $50 million.

Operational exposure is widespread in a quieter, less dramatic way. McKinsey's 2025 State of AI survey found that more than half of organizations using AI had experienced at least one negative consequence, with nearly a third reporting consequences specifically from AI inaccuracy. Inaccuracy, not breach. Just the model being wrong, at scale, in ways that cost something.

Against that backdrop, consider program maturity. EY's 2025 survey found that 23% of organizations still do not monitor vendor AI usage at all. Fewer than one in seven TPRM teams has reached what EY categorizes as fully matured automation capabilities. Most programs are still running on static, point-in-time questionnaires sent annually, a cadence designed for software that doesn't change between audits. AI models do change between audits, often materially, often silently, and often without any contractual obligation on the vendor's part to disclose it.

What Regulators Now Require From Organizations That Deploy Vendor AI

The most consequential regulatory development isn't any specific rule. It's a structural shift in where liability lands. Regulators have moved explicitly to holding the deployer responsible, not just the developer or vendor. If you're running a third-party AI system in a regulated context, "we bought it from a vendor" is no longer a defense.

The EU AI Act is the clearest expression of this. Article 26 imposes specific obligations on deployers of high-risk AI systems: use the system according to provider instructions, assign human oversight, ensure input data relevance, monitor ongoing operation, and report identified risks to providers and relevant authorities immediately. Logs must be retained for at least six months. Penalties for high-risk violations reach up to €15 million or 3% of global annual turnover.

The EU AI Omnibus, approved in June 2026, extends the core compliance deadline for certain Annex III systems to December 2027, but Article 26 deployer obligations remain fixed. The extension moves the enforcement date, not the substance of what's required.

DORA, in application across EU financial services since January 2025, treats AI vendors as ICT third-party service providers. Financial entities must maintain a register of contractual arrangements, monitor concentration risk, and demonstrate exit strategies. In November 2025, the European Supervisory Authorities designated 19 Critical ICT Third-Party Providers, including AWS, Google Cloud, Microsoft, Oracle, and SAP. Firms dependent on those providers must demonstrate assessed and mitigated concentration risk. Estimated compliance as of 2026 sits around 50% of financial institutions, with fines reaching 10% of annual turnover.

In the United States, the NIST AI Risk Management Framework is technically voluntary. But the FTC, CFPB, SEC, and EEOC all reference its principles in enforcement guidance, which makes it functionally significant in ways that "voluntary" doesn't quite capture. The Treasury's Financial Services AI Risk Management Framework, released in February 2026, maps 230 control objectives across the AI lifecycle with third-party risk explicitly addressed.

Sector and state-level requirements add further layers. FINRA's June 2024 Regulatory Notice 24-09 addressed GenAI and LLM use. Colorado's AI Act, effective February 2026, requires impact assessments and bias audits. New York City's bias audit requirement for AI hiring tools is already operative.

Legal precedent is reinforcing what regulation says. The Home365 settlement in May 2025, brought by the Pennsylvania AG, established that deploying a third-party AI tool and delegating accountability to the vendor is not a valid legal defense. The Workday lawsuit, allowed to proceed in federal court in May 2025, signals courts are willing to treat AI hiring tool deployers as liable parties.

You cannot assess a vendor once and treat the obligation as satisfied. The regulatory requirements are continuous, which means the liability is continuous.

Where Conventional Vendor Risk Assessments Break Down for AI Models

Standard TPRM was built on a coherent set of assumptions. Software behaves consistently. Security certifications attest to what matters. Annual reviews capture meaningful state. Questionnaires surface risks vendors haven't voluntarily disclosed. These assumptions held well enough for most of software history.

None of them hold reliably for AI vendors.

Point-in-time assessment versus continuous change: a model can be retrained between your annual review cycles. What you approved in January is running differently by June, with no contractual obligation on the vendor's part to notify you.

Security certifications don't address model behavior. SOC 2 attests to infrastructure controls. It says nothing about whether the model hallucinates, whether it has drifted, or whether it was trained on data that produces biased outputs. These are orthogonal dimensions that standard certifications simply weren't designed to evaluate.

Questionnaires depend on vendor disclosure. Vendors rarely volunteer their training data sources, model update policies, or known failure modes. There is no standard format for requesting this information, so you're asking open-ended questions and hoping for useful answers, which is a poor basis for risk management. One might argue that a thorough questionnaire is sufficient — but is it, if the vendor has no incentive to surface what they'd rather not disclose?

Sub-processor visibility is another gap. AI vendors typically sit on top of foundation model providers: OpenAI, Anthropic, Google, AWS Bedrock. That's a fourth-party dependency that standard two-tier TPRM doesn't capture. Your vendor has excellent security practices; their upstream base model provider's training data governance is a separate question entirely, and often an unanswered one.

Generic risk scoring misclassifies AI risk. A vendor with excellent security posture can still introduce severe model risk through bias, drift, or opaque outputs. Folding model risk into the security score misrepresents the actual exposure.

ISACA's 2025 guidance makes the break explicit: AI introduces hallucinations, model drift, and deep supply chain changes that require targeted assessment questions. Traditional models are no longer sufficient.

The Components a Dedicated Third-Party AI Model Assessment Needs to Cover

Pre-procurement is where most organizations have the most room to improve, because the questions that should be asked before signing a contract are still being asked infrequently, and the answers are rarely scrutinized with sufficient rigor.

Model provenance is foundational. What base model underlies the vendor product? Who trained it, on what data, and under what data governance terms? A vendor who can't answer these questions clearly is a vendor who doesn't know. That itself tells you something.

Update and retraining policy is where the non-determinism problem becomes contractual. How does the vendor notify customers of model changes? What is the re-validation process after an update? If there is no notification obligation in the contract, you need to negotiate one in before signing, not after.

Sub-processor and fourth-party mapping matters given how AI products are layered. Which foundation model providers and data processors sit underneath the vendor product? This is a direct line to your concentration risk exposure, and it's often the question vendors least want to answer in detail.

Explainability and auditability requirements vary by regulatory context, but it is also worth considering whether you can answer a regulator's questions today — not hypothetically, but with actual logs and decision trails the vendor can produce on demand.

Beyond pre-procurement, there are AI-specific risk dimensions a standard security review won't surface.

Bias and fairness: what testing has been performed, on what demographic groups, with what results? This is especially non-negotiable for AI used in employment, credit, housing, or healthcare contexts.

Hallucination rate and accuracy benchmarks: vendor-reported figures deserve methodological scrutiny. A model that performs well on an academic benchmark performs quite differently on the documents and queries your organization actually processes. Ask for validation on data that resembles your actual use case.

Prompt injection and adversarial robustness: what security testing has the vendor performed against LLM-specific attack surfaces? Most standard security questionnaires don't yet address this category.

Data handling in inference: does the vendor use customer inputs for retraining? Where are inputs logged? What are the possible exfiltration pathways?

The contractual provisions that typical vendor agreements omit are where practical protection actually lives. Model change notification obligations with defined notice periods. The right to audit or commission an independent third-party model evaluation. Data retention and deletion commitments that cover inference logs, not just at-rest storage. Liability allocation for harm driven by AI outputs. Exit strategy and data portability provisions, which DORA requires financial entities to demonstrate for critical ICT providers.

Not all vendor AI warrants the same depth of assessment. The criticality of the decision the AI informs, whether it's a credit decision, a hiring decision, a safety-critical operational output, or a content summary, should drive assessment depth. Data sensitivity is the other major axis. The EU AI Act's Annex III high-risk classification provides a regulatory basis for this tiering in EU-relevant contexts.

One framing note worth keeping in mind: the AI-specific framework supplements rather than replaces standard due diligence. Security controls, financial stability, and business continuity reviews still apply. The AI layer sits on top of them.

Why Ongoing Monitoring Matters as Much as the Initial Assessment

The initial assessment is a starting point. The core problem is that what you assessed at procurement can change without warning, and the vendor has no obligation, under most existing contracts, to tell you.

Model updates alter outputs. Safety filters are modified. System prompts embedded in vendor products can shift model behavior at scale, across every user of that product simultaneously. A point-in-time assessment has a half-life, and for actively developed AI vendor products, that half-life can be months. But how does this affect our original promise of knowing what you've deployed? It undermines it entirely, unless monitoring is continuous.

Continuous monitoring for AI vendors needs to track things conventional vendor monitoring doesn't. Model version changes and retraining events. Emerging CVEs or disclosed vulnerabilities in the model or its dependencies. Regulatory status changes, because a vendor's AI Act classification can shift as the product evolves. Incident and breach disclosures from the vendor or its sub-processors. And drift indicators: changes in output distributions or accuracy that surface in your own internal usage data, which is often the first signal that something has changed upstream before the vendor acknowledges it.

The regulatory basis for continuous monitoring is explicit. EU AI Act Article 26 requires deployers to monitor ongoing system operation and report identified risks immediately. Not periodically. On an ongoing basis. DORA requires financial entities to maintain and continuously update their register of contractual arrangements and demonstrate ongoing concentration risk management.

The organizational challenge is that, as EY's 2025 survey found, nearly a quarter of organizations still don't monitor vendor AI usage at all. Most programs that do monitor are doing it manually and reactively. Manual review cycles cannot match the pace at which vendor AI systems change. At enterprise scale, monitoring needs to be automated to be sustainable, which is a significant infrastructure investment that most programs haven't made yet.

How to Build the Organizational Structure to Sustain This Assessment Process

The ownership problem is where most programs stall. Third-party AI model risk sits at the intersection of TPRM, InfoSec, Privacy, Legal, and the business units that are actually procuring and deploying the tools. Without someone who explicitly owns it, assessments fall through the gaps between teams. Everyone assumes someone else is handling it.

The first structural decision is whether to extend an existing function or create dedicated ownership. Neither answer is universally correct; it depends on program maturity, organizational scale, and regulatory exposure. What matters is that the decision gets made explicitly, with authority and accountability that are unambiguous, not assumed.

Regardless of how ownership is structured, several organizational elements are necessary. A cross-functional working group with representation from TPRM, InfoSec, Privacy, Legal, and key business units creates the coordination surface for issues that span multiple domains. A centralized inventory of vendor AI products and their risk tiers is the prerequisite for any assessment or monitoring activity; you cannot assess what you haven't catalogued. Clear escalation paths for identified risks matter because AI-specific findings often don't fit neatly into existing risk registers or remediation workflows, and unclear escalation means issues sit unresolved.

Shadow AI presents a particular challenge because the procurement inventory is always incomplete. Effective ongoing monitoring requires mechanisms for surfacing tools being used outside sanctioned channels, through network monitoring, periodic self-reporting, or procurement policy enforcement. Usually some combination of all three.

The maturity trajectory worth aiming for is one where assessment is integrated into procurement workflows before contracts are signed, where monitoring is continuous and largely automated, and where the cross-functional structure produces consistent, repeatable decisions rather than ad hoc judgments under time pressure.

That's not where most programs are today. The gap between the risk level the data describes and the program maturity most organizations have achieved is still substantial. But the regulatory obligations are explicit, the cost of inaction is becoming easier to measure with each reporting cycle, and the structure of what's needed is legible enough that "we didn't know what to build" is becoming a harder position to defend.