GPAI upstream: what your foundation-model vendor owes you and what you must verify yourself

So today we are doing a deep dive into something that is frankly keeping a lot of tech executives awake at night right now Yeah, I mean it absolutely should be keeping them awake right because we have a massive stack of material in front of us We're looking at the finalized text of the EU AI Act We've got these really sweeping recent judgments from the UK High Court and this billion-dollar summary judgment out of California Plus we're pulling heavily from that really rigorous Operational framework from module 5 of the AI governance program exactly and the whole mission of our deep dive today is to take all this incredibly dense legal and technical documentation and You know figure out exactly how you as a developer or a company actually govern your relationship with the vendors who supply your AI models Because honestly, this is arguably the single most critical operational challenge in tech today I mean, we are officially out of the cute experimentation phase, right? Holy Sandboxes closed exactly. We were squarely in an era of strict unforgiving liability If you're building with AI today You are operating inside a highly regulated supply chain and to really feel the pressure of that supply chain I mean you just have to look at the trenches of the ongoing legal war right now Specifically the New York Times versus open AI and Microsoft lawsuit. Oh, yeah, that is just a brutal Grinding process right now.

It really is. I mean we're recording this in late 2026 and this lawsuit which was filed all the way back in December 2023 is still just sitting in this grueling wildly expensive Discovery phase in the Southern District of New York right in SDNY and discovery at that scale is just I mean It's a nightmare a complete nightmare In fact just a couple days ago on July 9th The Times filed this massive sanctions motion over exactly how open AI searched its own internal systems for copied material Which tells you everything you need to know about how impossibly complex these systems are. There's no final judgment yet There are no damages awarded It's literally just this highly public war over what exact data went into training a model, right? But here is where the reality sets in for you listening this you didn't build that massive model No, you're sitting one layer downstream from all of that exactly You are a developer or maybe a product manager who just bought access to a vendor's model through an API and you wired it directly into your own product you wired it into your internal workflow or you know a customer facing chat bot that processes live data every single day and if the federal judge in New York finally drops the hammer and orders that a foundation model has to be Fundamentally altered or even retrained from scratch because of copyright issues your downstream product does not get some polite grace period It just breaks it breaks immediately.

So before we dive into exactly how you protect yourself from that breakage We need to get our vocabulary straight. The law is very specific here Yeah terminology is everything because the law doesn't just regulate AI as some vague sci-fi concept It regulates specific actors doing specific things. So throughout this deep dive We're gonna use the term general-purpose AI model or GPI model How does the EU AI Act actually define that? So a GPI model is legally defined as a model that is trained on a broad quantity of data at scale it displays significant generality and It can competently perform a really wide range of distinct tasks regardless of how it's placed on the market Exactly the route to market doesn't matter for the definition Okay, and just to bridge the gap for the software engineers listening when we say GP AI model We are basically just talking about what the tech industry calls a foundation model, right? Yeah, that's the common industry shorthand a large general-purpose model trained on broad data Adapted to many downstream uses like the major large language models or image generators you use every day, right? So let's unpack the core vulnerability here because a lot of development teams live under what I call the illusion of outsourced risk Oh, this is such a dangerous mindset It really is a company signs an enterprise contract with a massive AI vendor, right? They get an API key they get this glossy Beautifully designed PDF called a model card and they treat those documents like a titanium shield, right? They assume that if the vendor doesn't explicitly mention a risk, you know Like copyright provenance or toxic data, then the model is just perfectly safe to deploy But that's a huge structural flaw in their thinking isn't it It's a massive flaw because simply put you cannot govern a vendors training data You literally can't do it because we don't have access to it.

Exactly. You don't have server access to their GPU clusters You didn't choose the petabytes of data. They scraped from the internet and you absolutely cannot audit their raw data lakes So trying to govern their internal engineering is just a total waste of time on a percent what you can govern and frankly what you? Are legally required to govern is the boundary between you and that vendor and in the AI governance framework They call this boundary the seam right? Yes the seam I really want to visualize this seam because it's not just a metaphor.

It is a rigid legal boundary It absolutely is so imagine you run a high-end restaurant. You don't raise your own cattle, right? You buy all your beef from a massive agricultural supplier. Okay, I like this analogy, right? So the supplier hands you all these certificates saying their slaughterhouses meet federal standards But then there's a huge outbreak I'm a massive recall on the beef because of severe contamination at the source farm, right? If you serve that contaminated beef to your diners pointing your finger the supplier and saying hey, we didn't raise the cattle That is not gonna stop your customers from getting violently ill No, and it definitely won't stop the Health Department from padlocking your front door Exactly the risk originated upstream But the liability actually materialized on your plate and the mechanics of that transfer are exactly the same in software The seam is that exact perimeter separating what the vendor legally owes you in writing? Versus what no vendor document is ever going to cover things You just have to verify yourself precisely because here is the foundational reason the seam exists The vendor documents the foundation model in general But you are responsible for the AI system in particular in particular, right which brings up this excuse I hear constantly from development teams.

Let me guess. We just call an API Yes, they look at the New York Times lawsuit and they go look we don't train models We just ping an API so these massive data lawsuits don't touch us We aren't the ones sitting in the defendant's chair, which is just a staggering misunderstanding of supply chain risk, right? I mean isn't being out of the courtroom enough to insulate you not even slightly look not being the defendant Just means you aren't paying the hourly rates for the trial lawyers in New York fair point It does absolutely nothing to remove the inherited operational risk If the foundation model your whole app relies on gets slammed with a court injunction You inherit that disruption the very second the judge signs the order your API calls just start timing out or worse the vendor panics and pushes a newly retrained compliant version of the model overnight and Suddenly all your carefully crafted pumps don't work anymore. Your system's accuracy just falls off a cliff Wow Okay, so knowing this massive cloud of risk is hanging over you upstream Doesn't really help until you figure out exactly where you stand in a blast radius, right? Exactly and the EU AI act treats you very differently depending on what you actually do with the model You can't demand what you're owed from a vendor until you know What legal shoes you're standing in right and your obligations flow entirely from your role and a really crucial detail that trips up a lot Of compliance officers is that this role is assigned per system not as some fixed company identity We really so a company isn't just one thing Nope, a single large enterprise could easily hold three different roles simultaneously across three different software projects Okay, let's clearly define these three roles because the source material makes it clear that beginners blur them constantly The first role is the GP AI model provider, right? The GP AI model provider is the organization that develops the general purpose model and actually places it on the market So this is the open AI the anthropic the Google exactly the massive capital intensive labs They secure billions in compute.

They scrape the web. They run the algorithms They make the model if you are listening to this you are almost certainly not occupying this role, right? You're the one buying from them. So stepping one layer down the supply chain What's the next role the second role is the downstream provider the act also calls this the provider of an AI system? This is the organization that builds an application Usually wiring up someone else's GP AI model at the core and then places that completed system on the market under its own name Okay So if my dev team takes a vendors language model Builds a nice custom UI add some legal domain prompting slaps our company logo on it and sells it as an AI Contract reviewer you are a downstream provider because we put our name on it Yes, because you place the finished system on the market you absorb the primary responsibility for ensuring it complies with the law before a user Ever touches it.

Okay, and the third rule is the deployer which has the lightest regulatory touch, right? Yes, a deployer is an organization that uses an AI system under its own authority in the course of its business But crucially without placing it on the market or substantially modifying it. Give me a real-world example of a deployer Sure, think of a regional hospital network They buy a fully baked off-the-shelf AI diagnostic tool from a medical software company They install it they use it to help their radiologists read x-rays, but they aren't selling it to other hospital exactly They didn't build it. They aren't selling it.

They're just using it that makes them a deployer now They still have legal duties human oversight logging informing patients, but they avoid the heavy manufacturing obligations of a provider But this boundary between deployer and downstream provider It seems incredibly fragile the source tech spends a lot of time Warning about this thing called the article 25 value chain trigger Oh, it is the most dangerous trapdoor in the whole EU AI act explain that because it sounds like a company could start out Intending to just be a deployer but then accidentally trigger article 25 and suddenly inherit a massive legal burden that happens all the time the law says that if you take an existing AI system and you put your own trademark on it or if you make a Substantial modification to how it operates or its intended purpose boom you legally become the provider of that system I can literally hear product managers across the world sweating right now. Yeah, because everyone is doing fine-tuning these days Yeah, so if a team says hey, we aren't rebuilding the model We're just doing some light fine-tuning on our own proprietary data to get better answers. We're still just a deployer, right? Can they use that defense that is a massive gamble because right now there is no bright Mathematical line in the case law that defines exactly what a substantial modification is.

It's a spectrum. Where's the line? Well, if you're just tweaking the temperature parameter on an API call to make the chatbot less creative You're likely still a deployer Okay But if you take an open source model and run a heavy fine-tuning process using thousands of specialized HR resumes So the model can now screen candidates, which it wasn't originally built to do exactly you have fundamentally altered its purpose You triggered article 25. You are now a downstream provider.

So what's the actual protocol when an engineering team is sitting in a meeting? Arguing over whether they're fine-tuning was substantial or not defensive pessimism. I love that phrase defensive pessimism It's the only way to survive the safe default if it's at all Ambiguous is to assume you carry the heavier role the downstream provider role until your external legal counsel Explicitly writes a memo telling you otherwise because if you guess wrong if you leave downstream provider duties unassigned Just because you crossed your fingers and hoped you were a deployer. That is a fatal compliance error You are pushing a high-risk system into production without doing the legal safety checks regulators will absolutely destroy you for that Okay, so let's assume a team does the hard work.

They map their systems They know their roles once you have that legal standing you finally have the leverage to turn around Face upstream and demand what the vendor actually owes you. Yes, and we aren't talking about polite customer service requests here We are talking about statutory obligations, right? And this is governed by article 53 of the EU AI Act, which officially became applicable on August 2 2025 correct GP AI model providers owe a finite highly specific list of four items It is not just whatever marketing brochure. They feel like emailing you.

Let's pull these four items apart The first one is the technical documentation of the model, which is under an exec AI But if I'm building an app this NX XI file isn't actually for me, is it? No, it's not annex XSI is strictly for regulators specifically the European AI office It requires the vendor to keep exhaustive highly confidential records of their engineering process their exact training methods Compute resources algorithmic architecture, right because that's full of trade secrets Open AI is not going to just hand over the exact architectural weights of GPT 5 to my startup Just because I paid 20 bucks a month for the API precisely. The IP risk is astronomical So as a downstream developer, you will almost never see the actual annex XI file But your due diligence must include getting formal written confirmation from the vendor that this documentation actually exists just another operating lawfully Exactly. Okay moving to the second obligation This one is explicitly designed for the people listening to this deep dive the downstream documentation, which is annex to all right Annex 12 is the information package engineered for you Its whole purpose is to give you enough info about the models capabilities and limitations so you can fulfill your own compliance duties But again, look at the scene here.

The law is balancing your need for transparency against the vendors IP The source material emphasizes that what you get in annex 12 is a described model That's a critical distinction. You get a comprehensive description use cases integration guidelines acceptable use policies general performance metric You don't get the raw training data You absolutely do not get the training data which transitions perfectly into the third obligation under article 53 And this is easily the most explosive legal minefield in the entire industry right now the copyright compliance policy Yes, article 53 paragraph 1 C It mandates that the GPA provider must have a formal policy to comply with EU copyright law And the absolute center of gravity here is that they must respect opt-outs for text and data mining which the industry calls TDM Let's pause and define Text and data mining because it sounds like a very clinical term for essentially vacuuming up the entire public Internet I mean functionally that is what it is TDM is the automated analytical process of scraping and processing Massive bodies of text images and code to extract patterns It's how they build the colossal training corpora these models need and the EU This is governed by article 4 paragraph 3 of the copyright directive of 2019 790, right exactly so how do these opt-outs actually work mechanically if I'm an independent artist or a news publisher and I don't want my life's work sucked into a foundation model How do I actually stop it under the copyright directive right shoulders have the legal authority to register explicit reservations of their rights? They say do not use my data for TDM Mechanically, this is mostly done through machine readable format like a robots dot txt file on a website, right? Or embedding specific metadata tags into digital images that signal to an AI crawler that the file is off limits Okay, but I need to underline a massive flashing warning sign here for the listeners The requirement is that the vendor must have a policy to respect opt-outs Having a policy document sitting in a corporate filing cabinet somewhere is not a legal certification that their actual training data is pristine and perfectly cleared The difference between a policy and a guarantee is the space where all your liability lives say that again That's huge the difference between a policy and a guarantee is where your liability lives Think of a chemical plant having a safety manual proves you have a process It does not prove an explosion didn't happen on the factory floor yesterday Wow, the vendor is promising the regulator. They try to respect opt-outs.

They are not indemnifying you against a copyright lawsuit Okay, that brings us to the fourth and final obligation under article 53 the public summary of training content Yes mandated by article 53 paragraph 1d The provider has to publish a detailed summary of the exact content they use to train the model and they have to follow a specific Template from the AI office. But again, look at the vocabulary a summary. It's a high-level overview.

I might say We use 30% English language websites 20% public domain books and 10% licensed journals, right? It is absolutely not a line-by-line URL by URL manifest of every single web page their crawlers ever hit it couldn't be a full manifest Anyway, the scale is incomprehensible. We're talking Trillions of tokens, but because it's only a broad summary It doesn't give you the deployer anywhere near enough granular visibility to verify if Specific copyrighted works were in there now a quick piece of operational intelligence for anyone doing vendor due diligence There's this general-purpose AI code of practice that was finalized on July 10th, 2025. Oh, this is a great tool, right? So if you're in a procurement meeting with an AI vendor asking them exactly which chapters of that code of practice Is an incredibly sharp revealing question because it forces them to move past the marketing speak and point to actual regulatory commitment Okay, I want to throw a massive wrench into this whole thing.

Well for it. There is this pervasive Stubborn rumor in the developer community right now regarding open source model. I know exactly what you're gonna say You've seen the forums, right? The prevailing wisdom is that if you use an open source model You know one where the weights and architecture are freely downloadable on hugging face That you are completely exempt from all this suffocating EU AI act red tape The community essentially thinks open source means open season.

It is one of the most dangerous widespread Misinterpretations of the law today. It's completely wrong mostly wrong, right? Look open source models specifically those released under genuinely free licenses where the parameters are public They do receive exemptions, but those exemptions are extremely narrow. What are they actually exempt from? They're only exempt from the first two obligations.

We just discussed They don't have to generate the annex xxi technical documentation for regulators And they do not have to provide the annex 12th downstream documentation to you Okay, so an open source lab doesn't have to spend hundreds of hours writing a corporate manual for developers Mmm, that makes sense. You don't want to crush open source innovation with admin overhead exactly But and this is the critical caveat they are not exempt from the copyright compliance policy They still have to respect TDM opt-outs and they are not exempt from publishing the public summary of training content Wait, what if that open source model is an absolute behemoth like as powerful as GPP for a clod 3? Ah, if an open source model is so massive that it crosses the threshold into the systemic risk tier Then every single exemption vanishes poof gone The open source provider has to comply with the entirety of article 53 just like a closed proprietary vendor And here is the most vital takeaway for you sitting downstream your own downstream obligations the testing the logging the risk management They do not get one ounce lighter just because you decide to download open weights instead of calling a paid API the fact that it's open source Changes the paperwork you receive from upstream It does absolutely nothing to dilute the fact that governing the deployed system is entirely your legal problem That is the perfect distillation of the seam you are responsible for the system regardless of the licensing model of the underlying components Okay, so we've established you have to be ruthless about checking what you actually receive against this four-part article 53 list But let's imagine a scenario where the vendor is flawless rare But okay, you buy access to one of the most powerful cutting-edge foundation models on earth The vendor hands you every document perfectly formatted does buying a heavily scrutinized super-powerful model somehow save you Does leaning on a massive top-tier vendor take the regulatory burden off your shoulders? It absolutely does not in fact it introduces a whole new layer of complexity and this brings us to the systemic risk tier Which is defined under article 51 of the Act How does the law define systemic risk because to me that sounds like a phrase a politician uses when a major bank is about to collapse it doesn't sound like a software definition the AI Act uses a very specific mathematical threshold a General purpose AI model is presumed to have systemic risk if it was trained using cumulative compute exceeding 10 to the 25th floating-point operations or FLOPs, let's translate FLOPs for the audience real quick a floating-point operation is basically a single fundamental Mathematical calculation right like adding or multiplying two decimal numbers It's the basic unit of computational effort for a processor, correct? And 10 to the 25th is a number so staggeringly large. It's almost impossible to comprehend It's a 1 followed by 25 zeros How much hardware is that practically to hit that level of compute during a single training run? You need tens of thousands of state-of-the-art GPUs running continuously for months consuming literally the electrical output of a small city We are talking about a very small Ultra-elite group of the absolute largest frontier models in the world The act also notes that the European Commission can independently designate a model as systemic risk Based on its market reach right even if it doesn't hit that exact math threshold Yes And if a vendor provides a model in this tier the law Imposes a heavy set of extra duties upon them under article 55 extra work like what under article 55 the provider has to perform Rigorous adversarial testing which we usually call red teaming Okay, they have to hire internal and external security experts to deliberately attack and try to break their own model Find vulnerabilities biases safety bypasses They also have to assess and mitigate systemic risks at the EU level Ensure incredibly high cybersecurity and report serious incidents directly to the AI office So from a downstream developers perspective, that sounds amazing.

If I license a systemic risk model I know the vendors just spend millions of dollars heavily red teaming it they attacked it from every angle So I can just staple their adversarial testing report in my compliance file and go to the pub Right doing that is falling directly into the model card trap the model card. Yes This extra upstream testing is incredibly useful intelligence for you. It tells you the base model is fundamentally robust But it is never under any circumstances a substitute for your own testing Let me use an analogy to explain the mechanics of why that is because this trips people up all the time relying solely on the vendors adversarial testing is Exactly like buying a heavy-duty commercial truck and just looking at its general crash test rating from the manufacturer, right? The manufacturer crashed that truck in a sterile climate controlled laboratory They drove it in a perfectly straight line and a flat concrete wall They prove the airbags deploy and the chassis holds up sir But they do not know that you just bought that truck Loaded with 10,000 pounds of shifting steel pipes and you plan to drive it through a treacherous icy mountain pass at midnight Exactly.

The manufacturer tested the vehicle in general. They absolutely did not test your specific highly dangerous delivery route that captures the dynamic perfectly a Model card even a highly detailed article 55 compliant model card for a systemic risk model It documents the model in a vacuum. It doesn't know your specific commercial context It doesn't know your intended purpose and it definitely doesn't know the specific demographic vulnerabilities of your end-users, right? The systemic risk provider proved the mathematical engine works safely in a general context just to satisfy the regulator But the intended purpose testing the risk classification of your app the design of your human oversight All of that is solely your responsibility The vendor proved the model is safe to exist You must prove the model is safe to use in your specific application and this profound Misunderstanding this false sense of security that because a big tech company tested the model You don't have to it leads directly to the single biggest vulnerability at the scene It is the issue that is keeping general counsel's across the world awake at night We need to talk about the origins of the training data This is where downstream teams crash headfirst into the floor versus clearance trap unpack that for us Well as we established under article 53 the vendor provides a public training summary and a copyright policy you have to ruthlessly train your teams to treat those documents as a floor a Basic baseline of operational info not as a legal clearance Let's define the two vital concepts that dictate this whole risk The first one is provenance provenance is simply the origin the lineage and the lawfulness of the data The model was trained on it's the pedigree of the data.

Did they buy it? Did they scrape it? Did they steal it and the second concept is regurgitation regurgitation happens when a foundation model if fed a very specific prompt reproduces exact Verbatim chunks of its original training data. It's not generating new text it's reciting memorized data exactly it could spit out a multi-page passage from a copyrighted sci-fi novel or a block of licensed proprietary computer code or even someone's private medical record that accidentally got swept up in the training corpus and Regurgitation is the absolute nightmare scenario because it takes an upstream provenance issue data The vendor scraped three years ago and it turns it into a downstream Emission problem right under your company's name right on your user interface and the source text points out that this provenance risk Reaches your product in three distinct highly damaging ways Yes the first is legal disruption that travels meaning what if an upstream foundation model is suddenly found by a court to be trained on Stolen data and the judge orders the model pulled offline your downstream product breaks instantly The risk travels right through the API the second way it hits you is the regurgitation We just defined which brings direct strict liability right to your doorstep If your chatbot spits out a New York Times article verbatim to a user the Times might sue the model maker But they can absolutely sue you for distributing it Oh without a doubt and the third way provenance risk manifests is what we call the quiet poison in the corpus quiet poison That is a terrifying phrase explain the mechanics of that Well provenance isn't solely about copyright and fair use The sheer volume of data scraped from the open internet virtually guarantees that the training corpus contains material that is outright illegal Highly toxic or just deeply biased give me a historical example of this actually happening because this isn't theoretical not at all in 2023 there was this massive widely used public image training data set called la ion It was instrumental in training several major open source image generation models, right? Researchers eventually discovered and publicized that the data set contains suspected links to illegal child sexual abuse material Yeah, the data set had to be immediately withdrawn So as a downstream deployer you could be inadvertently relying on a model built on incredibly toxic Radioactive foundations and you would have absolutely no idea because the article 53 public summary of training content just says we used internet Images exactly. It doesn't tell you the images were illegal You're operating completely blind and to make matters worse the legal landscape governing copyright provenance is Incredibly fragmented and volatile.

It's a complete patchwork of jurisdictional contradictions completely Let's look at the actual case law to prove this because the UK High Court handed down a massive decision on November 4th 2025 in the case of Getty Images versus stability AI Getty sued stability claiming they unlawfully used Getty's massive copyrighted photo library to train their image generation model and The outcome of that case absolutely shocked many people in the creative industries The UK High Court actually rejected the central copyright infringement claim. Wait, they rejected it They did the court accepted the technical argument that the fully trained foundation model did not contain actual stored copies of Getty's photographs The model only contained mathematical representations of the patterns within those photographs So the model maker won the big copyright argument, but Getty didn't walk away completely empty-handed did they? No, they didn't the court did find stability AI liable for historic trademark infringement, right? Because early versions of the model were actually regurgitating distorted versions of the famous Getty watermark Directly onto the generated images which confused consumers exactly But the core lesson here is that in the UK at least in that specific Jurisdiction on that specific day the act of training on copyrighted images was not deemed a copyright violation in itself Which perfectly illustrates the cross-border jurisdictional differences you have to navigate what is considered lawful protected innovation in London Might be considered massive systemic theft in New York Yes and to prove that point We just have to look at the United States and a case that I think provides the most vital lesson of this entire deep-dive Barts versus anthropic. This is a massive case huge This is a class-action lawsuit filed by authors who alleged that anthropic trained its clawed language models on Thousands of pirated books and we have a fascinating truly surgical summary judgment from June 2025 by Judge William Alsop in the Northern District of California Right and Judge Alsop drew a razor-sharp legal and economic line here He ruled that if an AI company lawfully purchased books and used them to train a model that action was Quintessentially transformative and protected under the US doctrine of fair use so buying the data is legally protected innovation But what was the other side of that line? The other side of the line was the acquisition method The judge ruled that downloading and training on pirated copies of books from illegal shadow Libraries was absolutely not fair use because it's stolen property Exactly the economic theory being that utilizing stolen property to build a commercial product Inherently undercuts the market for the original work and the financial outcome of that distinction was absolutely staggering in September 2025 Anthropic agreed to a settlement paying a minimum of 1.5 billion dollars That broke down to roughly three thousand dollars per pirated work and Crucially, they were forced to destroy the pirated copies within their training data sets That case is a watershed moment for AI governance But let me play devil's advocate for a minute if a major wildly well-funded AI lab just settled for a billion and a half Dollars over pirated books doesn't that just signal to the market that the whole concept of fair use is dead Shouldn't downstream deployers just assume all upstream training data is completely toxic.

No, that is the wrong lesson to draw entirely What the Bart's case proves mechanically is that how the data was acquired? Matters just as much if not more than the abstract legal concept of fair use how you acquired it Exactly the exact same text used for the exact same algorithmic training purpose was protected fair use if the vendor bought it But it was billion-dollar infringement if they pirated it and here is the terrifying part for you sitting downstream building an app You cannot see the acquisition method. No, the vendors summary of training content will tell you they trained on English language literature You will not tell you if they paid for the literature or downloaded it from a Russian shadow library Which means a deployer cannot simply ask a vendor is your training data lawful? The vendor will always say yes every time of course, they will you have to ask explicitly exactly how did you acquire the rights to this specific corpus of data because as the Bart's case proves a Vendor's internal definition of lawful might not survive contact with a federal judge Wow And because you cannot blindly trust the vendors data and because you can't rely on the vague assurances in their copyright policy You have to ruthlessly interrogate every single claim They make about their model not just regarding safety and copyright, but regarding its actual operational capabilities Yes, and this brings us to the operational discipline of decomposing claims to defend your product decomposing claims Look vendors are fundamentally sales organizations. They love to sell with definitive glossy numbers They will hand you a model card that boldly claims our model achieved state-of-the-art 98% accuracy on financial reasoning or we have driven our hallucination rate down to below point zero one percent and for a stressed Compliance officer or a busy product manager It is so incredibly tempting to just copy those numbers paste them directly into your own internal compliance Documentation and competently present them to your board of directors Oh people do it every day and doing that is a profound governance failure.

You are inheriting a liability. You do not understand So what's the rule you must institute a strict internal rule? You refuse to inherit a number that you cannot decompose How do you actually decompose a claim mechanically any? Capability score any safety benchmark any hallucination rate presented by a vendor must be broken down by asking three unforgiving questions First what exactly was measured second on what specific data set wasn't measured and third who judged the outcome? Okay, let me use an analogy here because taking a vendor's hallucination rate at face value is exactly like signing a commercial loan application Using an income figure that a mortgage broker just scribbled in for you without you ever seeing the underlying pay stubs, right? If the bank comes knocking a year later because the numbers were entirely fabricated You can't just point at the broker and say well, they wrote it down on a glossy piece of paper You signed the document you deployed the system You're the one on the hook for the fraud the data bears that out entirely if a vendor says they have 99% accuracy But when you ask on what data they reveal they only tested it on basic middle school math problems Well that 99% figure is completely useless to your enterprise financial software Obviously if you cannot answer what was measured on what data and by whom you cannot label it as a fact So what you call it? You must strictly label it in your conformity files as an unverified assertion by the vendor Unverified assertion. Yes And once you identify an unverified assertion or any open risk like copyright provenance You have to route it to a solution in the governance framework.

You route every single open risk to one of two paths There is no middle ground right the source material says every open risk must go to test or contract Nothing's allowed to float unassigned exactly Let's break down those two buckets starting with test What actually goes into the test bucket the test bucket is reserved for claims and risks that you can physically Mechanically verify at your own desk You mean example you test for accuracy by running the model against your own proprietary golden data sets that represent your specific business domain You test for regurgitation by hitting the model with adversarial prompts designed to mirror what your actual users will type You test to see if the vendors documented limitations in annex 12 actually match how the model behaves one wired into your specific infrastructure Okay And if a risk cannot be tested at a desk Like the risk of a massive copyright lawsuit over the vendors training data you read it to the contract bucket, correct? You contract for risks that turn entirely on vendor conduct or on unsettled volatile areas of the law You can't rate a Python script to test away the risk of a copyright lawsuit So you route that risk to the procurement team you demand a robust contractual indemnity for third-party intellectual property claims You ask for explicit contractual warranties detailing exactly how the training data was acquired You demand contractual notice periods forcing them to give you 90 days warning before they deprecate a model version, but wait We need to clear up a massive misconception about contracts here because I've had executives tell me straight to my face We don't need to worry about copyright provenance or toxic data because we negotiated a bulletproof indemnity clause in our enterprise agreement Doesn't an indemnity just make the risk disappear? Like if the vendor promises to pay my legal fees the problem is solved, right? That is such a dangerous oversimplification of contract law and Indemnity does not stop the rain. It merely dictates who pays for the umbrella when the storm inevitably hits It allocates the financial liability after the damage is already done Exactly. It absolutely does not replace the regulatory need to test the leaks You could have fixed yourself consider the mechanics of reputational damage if your AI system regurgitates sensitive personal data and Catastrophically harms a user the vendors indemnity might cover the cost of the ensuing civil lawsuit Yeah, but that indemnity will not repair your company's shattered reputation in the market No, it won't and it certainly won't stop the European privacy regulator from knocking on your door and finding you for failing to implement Basic technical safeguards you cannot contract your way out of regulatory negligence You need both rigorous testing and robust contracts which leads us to an area where deployers often exhibit a complete blind spot Because we have spent this entire deep dive talking about the rain coming down from the vendor though Legal and technical risks flowing downstream into your product, right? But the scene we've been discussing is not a one-way street Is it there is a massive flow of data and risk moving in the exact opposite direction? Yes, we are talking about the reverse flow the reverse flow mechanically define that for us The reverse flow is the data your system sends up to the vendors model via the API payload on every single interaction Let's make this highly tangible Every time your customer types a query into your customer service chat bot or every time your employee highlights a sensitive internal strategy document and ask the AI to summarize it that text is packaged into an API call and Sent upstream to the vendors servers and the critical question is what happens to that data? Once it reaches the vendor, right? The answer depends entirely on the specific terms of service you agreed to and the specific technical configuration of your API account The vendor might log those prompts for quality assurance They might retain that data on their servers for 30 days or maybe six months Right and most dangerously if you are using a standard commercial tier They might actively use your proprietary inputs to train their future foundation models The ultimate cautionary tale for reverse flow is the famous 2023 Samsung case.

Oh, it is the textbook example of reverse flow failure. What exactly happened? Engineers at Samsung were struggling with some complex highly confidential proprietary source code and looking for a shortcut they pasted the secret code directly into a public consumer tier AI chat bot and Asked it to debug the software and the tragedy here is that nobody hacked Samsung no cyber criminal breached their firewalls Their own highly cleared employees operating in the course of a normal workday Voluntarily shipped the company's deepest intellectual property upstream to a third-party vendors servers Because they didn't understand how the model ingested data They forced Samsung to implement sweeping draconian bans on the internal use of generative AI. It was a disaster So hearing that a lot of IT directors will say look that's an easy fix We just tell our cloud administrators to flip the configuration toggle in the API settings from consumer to enterprise Which opts us out of data training problem solved Is it really just an IT toggle treating reverse flow as a simple IT toggle ignores the massive legal reality beneath it? If the data you are sending upstream in those API payloads includes any personal data from your European users names addresses behavioral data You have instantly created a strict highly regulated legal relationship under the general data protection regulation the GDPR How does the GDPR actually govern that API call under article 28 of the GDPR? You are legally required to execute a data processing agreement or DPA with the vendor a DPA Yes, this is not a technical configuration It is a binding legal contract that dictates exactly who answers first if that personal data is mishandled or breached So it defines the roles exactly the DPA establishes whether the vendor is acting merely as a processor of your data Meaning they follow only your strict instructions or if they are acting as a joint controller meaning they have a say in how the data Is used governing the reverse flow is a massive complex Compliance surface.

Okay. Let's take a breath and survey the landscape here. You've mapped your roles.

You've audited the article 53 documents You've contracted for indemnities regarding copyright provenance. You've physically tested your prompts to prevent regurgitation You've secured your reverse flow with a bulletproof DPA. Your system is locked down and safe Yeah, you can finally relax really you can relax only until the vendor decides to change the laws of physics overnight This brings us to the ultimate operational headache Model version change the fundamental truth of API driven AI is That the foundation model you rigorously validated on a Tuesday is not necessarily the exact same model You are running in production on a Friday.

Wait, they just change it. Oh all the time vendors are constantly silently updating weights retraining algorithms to patch vulnerabilities and Deprecating older models to save on compute costs so you can spend a month Building a flawless conformity file documenting all your rigorous testing based on version 1.0 of a model but that entire file becomes completely useless fiction if the vendor silently updates your API endpoint and moves you to version 1.1 behind the scenes precisely to use an analogy Validating a model once and assuming it stays static is like trusting a map of a coastline that was drawn five years ago Fully knowing that the tide and the storms are constantly washing away sand and reshaping the shore, right? The map might have been perfectly accurate the day the cartographer drew it But if you try to sail a commercial ship by it today You are going to crash into a hidden reef and the most important part of that analogy is liability If your ship crashes because your map is out of date and your passengers get hurt. They are gonna sue the ship's captain You they are not gonna sue the cartographer who drew the map five years ago.

We have actual Finalized case law proving this exact dynamic. Oh, we do Look at the Moffitt versus Air Canada case, which was decided by the British Columbia Civil Resolution Tribunal on February 14th 2024 walk us through the mechanics of that case. So a customer.

Mr. Moffitt went to Air Canada's website Seeking a discount for a bereavement flight after a death in his family He interacted with the airline's automated AI chat bot the chat bot hallucinated it confidently gave Mr. Moffitt completely fabricated information regarding the airline's refund policies Relying on the chat bots advice. He purchased full price tickets expecting a retroactive refund that did not actually exist And when he realized he was out hundreds of dollars, he sued it was the airline's defense Air Canada attempted a truly audacious legal defense They tried to argue that the chat bot was essentially a separate legal entity Responsible for its own words and that the customer should have double-checked the chat boss answers against the airline's static policy pages How did the tribunal react to the idea that an AI is responsible for its own hallucinations? They rejected the argument completely. They ruled the while a chat bot is an interactive component It is legally no different from any other piece of information on a company's website They held the airline fully financially liable for the negligent misrepresentation The core inescapable principle here is that the party facing the customer carries the exposure The customer holds you responsible for the failure not the unseen AI vendor whose language model generated the text That is the ultimate reality of the scene The vendors article 53 duties run upstream to the European regulators and they run horizontally to you as the buyer But those vendor duties do not stand between you and the customer your system harms Which means you have to treat model version as a first-class operational fact in your code base you cannot just point your system to a Generic API endpoint like latest model.

You have to pin it. You must explicitly pin the specific Alphanumeric version of the model you are using in your code You must implement automated monitoring to detect if the models behavior drifts you must contractually demand 90-day notice periods before a vendor deprecates a model and Critically you have to rerun your most vital test suites on every single new version You are moved to before you allow that new version to face your customers Governing an AI system is not a one-time sorting task. It is a continuous operational discipline bringing all this theory together Yeah, I mean defining your roles demanding the for article 53 documents investigating copyright provenance Decomposing unverified claims securing your reverse flow and pinning your model versions.

It can feel overwhelming It's a lot. It requires a single unified artifact You need something tangible something you can build immediately to actually govern this seam in the real world Which brings us to the most valuable move you can make when you walk into the office on Monday morning Everything we've discussed converges on one single executable document. It is called the vendor Documentation review walk us through exactly how to build this blueprint You start by identifying your highest stakes highest risk AI system currently in development or production You open a document and create a simple two column record.

What goes in the left column? The left column is your owed and received column This is where you map exactly what the vendor legally owes you under article 53 against what you physically have in your possession Okay, so checking the boxes exactly. Do you have written confirmation of the annex effect technical documentation? Do you have the annex except downstream documentation describing the models limits? Do you have their copyright policy respecting TDM opt-outs? Do you have the public summary of training content on the AI office template? You ruthlessly check off what the law says you're owed versus what the vendor actually delivered It's a basic inventory check, but the real governance magic happens in the right column Exactly. The right column is your must verify yourself column This is where you list every single risk that the vendor documents explicitly do not cover like provenance You list copyright provenance you list regurgitation of sensitive data You list accuracy on your highly specific domain pass you list reverse flow data leaks You list silent version changes and for every single item in this right column You must explicitly route it to either test or contract no floating risks You write down exactly the data set you will use to test it or the exact Indemnity warranty you are demanding from the procurement team Nothing is allowed to float unassigned and for the final polish on this document You write a candid one paragraph boundary statement.

You place this at the very top of the review It must state clearly what your legal role is what risks the vendor covers what risks they do not cover and it must name a specific Residual risk that your company is accepting. It should be honest Credible and leave zero filler if you hand that document to a board of directors Or if you hand it to an investigating regulator, they will instantly see the mechanics of your oversight They will see that you are actively governing the system. Not just closing your eyes and hoping for the best It transforms the weak excuse of we just use a vendors model from a helpless shrug into a defensible legally governed position Before we wrap up I want to leave you in the room with one final Provocative thought to mull over as you audit your own AI supply chain this week.

This is a good one Think very carefully about the middlemen Are you actually buying your foundation model direct from the GPI model maker? Or are you buying through a massive cloud marketplace a third-party API broker or an enterprise software platform? That simply wraps someone else's foundation model in a nice interface because the law demands Documentation from the creator a reseller cannot hand you article 53 annex 12 documents that they never actually authored exactly, if your legal compliance depends entirely on tracing documentation back to the original source a Reseller is very often a dead end that breaks your compliance chain So next time you log into your developer dashboard and look at your API keys ask yourself Do you actually know who is at the other end of the line? It is a question every deployer needs to answer before the regulator asks it for them We'll catch you next time on the deep dive. Stay curious and govern your seams