The license to govern: explaining to a skeptic exactly how your model fails
The short answer
The failure account is the license
The deliverable of this topic, the "how my model fails" explanation, is what converts hands-on building into governance authority. You do not earn the right to govern a system by reading about it; you earn it by being able to explain, to a skeptic, exactly how the system you built breaks.
What you will be able to do
- Analyze your own trained model into a structured set of named failure modes, each with the conditions that trigger it, the signal that reveals it, and the consequence if it reaches a user.
- Distinguish the kinds of failure an AI system produces (fabrication, variance, distribution shift, brittleness, silent error, miscalibrated confidence) and explain why each demands a different governance response.
- Construct a "how my model fails" explanation in plain language that survives a skeptic who wants it to be wrong.
- Separate what you know about your model's failures from what you do not yet know, and record the unknowns honestly rather than papering over them.
- Explain why a demonstration that "it worked when I tried it" is not evidence of reliability, using the non-determinism you observed in Topic 1.4.
- Justify the ordering law of this program (build before you govern) by showing that governance authority comes from having done the work, not from having read about it.
- Map each failure mode you name to the later point in the program where that failure must be measured, contained, or documented for a regulator.
- Size each failure mode by severity, likelihood, and detectability, and identify the modes that are severe, frequent, and silent enough to demand a mandatory human checkpoint.
- Assign a named owner and an explicit decision (mitigate, accept, or block) to every open failure mode, so that no documented risk drifts into production unowned.
The ideas, one by one
Specificity is the whole value
A failure account names distinct modes, each with what it fails at, its trigger, its detection signal, its measured frequency, its consequence, and its mitigation. "The model may sometimes be wrong" is a disclaimer, not an analysis, and a governor can do nothing with it.
Write for the hostile reader
The people who will read your governance work are auditors, regulators, lawyers, and hostile boards. A document that satisfies a friendly reader is worthless. The bar is a skeptic who wants it to be wrong and cannot find a hole you did not already name.
One run is not evidence
Because of variance, brittleness, and the demo effect, a single successful run tells you the system can succeed, not that it will. Reliability is a property of the distribution of outputs. Every frequency in your account comes from repeated runs, or it is honestly marked "not yet measured."
Name the unknown
An expert account separates what you have characterized, what you know but have not measured, and what you have not looked at. The "what I have not tested" section is never empty. Naming your blind spots protects your credibility and becomes the test plan for later evaluation work.
Honesty is the ethics
Overclaiming a system's reliability is not a technical slip; it exposes users to harms you chose not to name. The first governance act is an honest failure account. Refusing to write one, or writing a dishonest one, is the first governance failure.
Different modes, different responses
Fabrication, bias, drift, and brittleness are distinct failure modes that demand distinct governance responses (grounding, disaggregated measurement, monitoring, robustness testing). Collapsing them into "errors" destroys the analysis. Analyze means separate.
The document is living
Your account carries a date and a model version and is updated the moment the model, the data, or the world changes. The versioned, maintained form is developed in Topic 10.3. A static failure account is stale on arrival.
It grounds everything after
This artifact is not filed and forgotten. The eval suite in Module 4 must catch the modes you named; the conformity file in Module 5 lifts its honest core; the hostile board in Module 8 is survivable because you wrote your weaknesses down first. Module 1 exists so that the rest of the program stands on something real.
Size before you prioritize
Naming failures is half the analysis; sizing them by severity, likelihood, and detectability is the other half. The mode that is severe, frequent, and silent is the one that reaches users, because nothing was designed to stop it. Sizing tells a finite testing budget what to test first and tells the oversight design where a human checkpoint is mandatory.
The method is old, the modes are new
You are applying decades-old safety-engineering discipline (Failure Mode and Effects Analysis) to a class of system whose failures (fabrication, variance, silent error) are unfamiliar. The honesty requirement is identical to any safety-critical field; what is new is that AI fails confidently and quietly, which raises the premium on naming the failures in advance.
Listen to it
The same lesson, talked through by two hosts as episode 10 of the podcast.
Real cases
These are real, documented practices and events that show the discipline of explaining how a system fails, done well and done badly. The Cursor case anchors the immersive scenario in Section 5; these show the wider pattern.
Example 1: Model cards as a documented practice (Mitchell et al., 2019). In 2019 a team of researchers including Margaret Mitchell and Timnit Gebru published "Model Cards for Model Reporting," proposing that every trained model ship with a short document stating its intended use, its performance broken down across groups and conditions, and, critically, its known limitations and the situations where it should not be used. The proposal has become common practice at major AI developers. The lesson for your failure account is the structure: a model card is not a sales sheet. Its most valuable section is the honest limitations section, which is exactly the "how my model fails" explanation you are learning to write. A model card without a real limitations section is a brochure.
Example 2: Frontier system cards and honest limitation disclosure. When large AI labs release flagship models they now commonly publish a "system card" or "model card" that documents known failure modes: tendencies to hallucinate, susceptibility to jailbreaks, overreliance risks, and areas of weak performance. OpenAI's GPT-4 System Card (OpenAI, March 2023) devoted substantial space to documented risks and limitations rather than capabilities alone. The teachable point, marked as established practice: the credibility of these documents rises with the specificity of the failures they admit. A system card that says "the model may occasionally make mistakes" has told you nothing; one that names the categories of mistake, the conditions, and the mitigations has given you something to govern with.
Example 3: NIST AI RMF and the Measure function (NIST AI 100-1, 2023). The United States National Institute of Standards and Technology published the AI Risk Management Framework (AI RMF 1.0) in January 2023, with four core functions: Govern, Map, Measure, and Manage. The Measure function is, in effect, an instruction to characterize how your system fails: identify risks, analyze them, and track them. The framework is voluntary, not law, and that is the point of citing it here (established): the discipline of documenting failure is treated as foundational risk management across the field, not as an optional extra. Your failure account is the raw material the Measure function operates on.
Example 4: FMEA in safety-critical engineering. Failure Mode and Effects Analysis has been standard practice in aerospace, automotive, and medical-device engineering for decades. Engineers sit down before a system ships and enumerate, part by part, every way it can fail, how severe each failure is, how likely, and how detectable, producing a risk priority for each. The relevance to AI governance is direct and the transfer is the lesson: you are applying a mature safety-engineering discipline to a new class of system whose failure modes (fabrication, variance, silent error) are unfamiliar to engineers trained on mechanical parts. The method is old; the modes are new; the honesty requirement is identical.
Example 5: The legal duty to disclose limitations (context, verify per jurisdiction). Regulators are moving from recommending to requiring that AI providers document known limitations and foreseeable misuse. Under the European Union (EU) AI Act (Regulation (EU) 2024/1689), high-risk providers will be required to supply instructions for use that describe the system's capabilities and its known and foreseeable limitations; that specific obligation is part of the high-risk rules, whose application has been deferred (stand-alone high-risk systems to 2 December 2027, high-risk AI embedded in regulated products to 2 August 2028), while the Act's prohibited-practices and AI-literacy duties are already in force. The deep treatment of these obligations, and their current timeline, belongs to Module 5 (see Topic 5.6); the point here (established, with the application date deferred) is that a "how my model fails" explanation is not only good practice, it is becoming a legal deliverable on a fixed clock. The honest account you learn to write in this topic is becoming the thing a regulator will eventually be able to demand to see.
Example 6: Datasheets for datasets, the same discipline one layer up (Gebru et al., 2018 onward). A parallel proposal, "Datasheets for Datasets" (Timnit Gebru and colleagues, first circulated 2018), argued that every dataset should ship with a document stating how it was collected, what it does and does not represent, and its known gaps and biases. The relevance here (established practice): most of your model's failures are downstream of its data, so an honest failure account and an honest data document are two views of the same truth. The data-document form is developed when you build a data provenance file in Module 2 (see Topic 2.6); the point in this topic is that the discipline of honest limitation-documentation runs through the whole stack, from data to model to deployed system, and each layer's honest account feeds the next.
Example 7: The cost of not having one (the Cursor pattern, generalized). The Cursor support-bot incident (The Register, April 2025) is the anchor of this topic's scenario, so it is treated in depth in Section 5. Generalized, it is the counter-example to all of the above: a deployed AI system whose failure mode (confident fabrication of policy) was foreseeable, was a known category of LLM behavior, and yet was apparently never written down as "here is how this bot will fail and what it will cost us." The absence of the document is the failure behind the failure. Its remedy, notably, was partly the discipline this topic teaches: after the incident the company began labeling AI-generated support responses, which is a detectability control aimed squarely at the silent-fabrication failure mode. Every example above is a version of the same claim: the organizations that survive their AI's failures are the ones that named the failures first.
Where people go wrong
- "If I list all the ways it fails, people will lose confidence in the system." Backwards. A specific, honest failure account raises confidence among the people whose confidence matters, because it proves you understand the system. The people who lose confidence are the ones who were relying on you not to look. A skeptic trusts a document that admits real limits far more than one that claims none. Overclaiming is the thing that destroys credibility, not honest disclosure.
- "It worked when I tested it, so it works." The single most dangerous inference in AI, addressed in full in Section 3E. One successful run tells you the system can succeed on that input on that run. It does not tell you it will succeed, because of variance, brittleness, and the demo effect. Reliability lives in the distribution of outputs, not in the one output you saw.
- "The failure account should only list failures I can fix." No. Some of the most important entries are failures you cannot yet fix, or cannot fix without breaking something else. Hiding an unfixed failure because it is embarrassing is how the Cursor pattern happens. The account documents reality, including the reality that some risks are open. The decision to ship with an open risk belongs to a named human, and the document is what makes that decision visible.
- "A high-level statement of limitations is enough." "The model may sometimes be inaccurate" is not a failure account; it is a disclaimer. It names no mode, no trigger, no frequency, no consequence, and it gives a governor nothing to act on. Specificity is the entire value. If a later topic cannot build a test from your entry, the entry is too vague.
- "Fabrication and bias and drift are all just 'errors,' so one line covers them." Each is a distinct failure mode demanding a different response. Fabrication needs grounding and external checking. Bias needs disaggregated measurement across groups. Drift needs ongoing monitoring against fresh data. Collapsing them into "errors" destroys exactly the analysis the topic is asking you to do. Analyze means separate, not lump.
- "I have to know the exact frequency of every failure." You do not, and pretending you do is the mistake. Where you have measured a rate from repeated runs, report it with its source. Where you have not, write "not yet measured" and flag it. An honest "not yet measured" is a legitimate entry; an invented number is a falsification that a skeptic will eventually expose.
- "Governance is a separate job from building, so I do not need to have built the thing." This is the exact belief this program exists to refute. The whole ordering law, build before you govern, says your authority to govern comes from having done the work. A failure account written by someone who has never made the model fail is a guess. Yours is grounded, because Topics 1.1 through 1.5 put your hands on the system.
- "Once I write the failure account, I am done with it." The document is living, not final. Colin adds a line the same evening his account misses a gap. The versioned, maintained form of this document is the subject of Topic 10.3 (see Topic 10.3). A failure account with a date and no updates is stale the moment the model, the data, or the world changes.
- "Every failure mode is equally important, so I list them and stop." An unsized account is half-analyzed. Without severity, likelihood, and detectability for each mode, a reader cannot tell the trivial from the catastrophic, and a finite testing budget gets spread evenly across failures that deserve very different attention. Section 3H is the sizing step; skipping it leaves the account descriptive rather than actionable.
- "The rare catastrophic failure is not worth documenting because it almost never happens." This is the most seductive omission, because dropping the rare severe mode makes the account shorter and calmer. It is also the omission that leaves the worst outcomes undocumented. Low likelihood does not lower severity; a severe, rare, low-detectability failure belongs in the account precisely because its rarity makes it easy to forget and its severity makes forgetting it unacceptable.
- "An AI tool can write my failure account for me." An AI is a useful red-team generator (Section 8) and a useful enumerator of candidate modes, but the account's authority comes from grounding in behavior you observed yourself. An AI-written account substitutes generic plausible risks for your system's real, measured ones and cannot report the frequencies only your runs can produce. Use the AI to attack and prompt your draft; do not let it be the source of the facts.
Questions people ask
- What is "How my model fails" explanation (the failure account)?
- The structured, plain-language document produced in this topic that names each way a specific model fails, with, for each mode, what it fails at, its trigger conditions, its detection signal, its measured frequency, its consequence, and its current mitigation, plus a statement of intended use and an explicit list of what has not been tested. It is the artifact that grounds the learner's governance authority and is consumed by the evaluation report (Module 4) and the conformity file (Module 5).
- What is failure mode?
- A distinct way a system produces a bad output. Enumerating failure modes, rather than describing failure in general, is the core analytical act of this topic and is borrowed from safety engineering's Failure Mode and Effects Analysis. More on Failure mode
- What is Failure Mode and Effects Analysis (FMEA)?
- A structured safety-engineering method, long standard in aerospace, automotive, and medical-device work, for listing every way a system can fail, how likely, how detectable, and how severe each failure is. Applied to AI, the method is unchanged but the failure modes (fabrication, variance, silent error) are new. More on Failure Mode and Effects Analysis (FMEA)
- What is fabrication (confabulation, hallucination)?
- A failure mode in which the model produces fluent, confident, false output that looks indistinguishable from a correct output. Named as a distinct generative-AI risk in NIST's Generative AI Profile (NIST AI 600-1, 2024). The Cursor support bot inventing a device policy is a fabrication.
- What is variance (non-determinism)?
- A failure mode in which the same input produces different outputs on different runs. Variance means a single test result cannot establish reliability, because passing once does not guarantee passing again.
Keep going
This lesson builds Model failure modes and bias recognition, and that page shows the roles that hire for it. Every Certified AI Governance Professional (CAIGP) lesson.
Watch the lesson and prove it
Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.
Watch the lesson and prove it →