Building the eval suite that would have caught your Module 3 incident
The short answer
An eval suite is not a demo, a unit test, or a benchmark
Only a suite, with fixed inputs, stated expected behavior, and a reproducible count, can answer "would this have caught our incident?" The other three cannot, and confusing them is how undefended numbers get shipped.
What you will be able to do
- Convert a real incident from your Module 3 dossier into a concrete, runnable evaluation case with a fixed input and an expected behavior.
- Distinguish an evaluation suite from a demo, a unit test, and a benchmark, and explain why only the suite gives you a defensible number.
- Define a metric precisely enough that a stranger could reproduce your number: what counts, on which inputs, judged by whom.
- Build a small golden set (a fixed collection of inputs with known correct answers) that includes the incident class, adversarial cases, and ordinary cases.
- Choose a grader for each case (exact match, rule check, human review, or a model-as-judge) and state that grader's failure modes.
- Set a pass threshold and a regression gate so a known failure cannot silently return in a later version.
- Explain the honest limits of your suite, so you never repeat the Pieces claim of a precise rate you cannot defend.
The ideas, one by one
Build the suite from the failure you already have
Your Module 3 incident is the seed case, entered verbatim. You are not guessing what might go wrong; you shipped what went wrong. The incident case is the spine the whole suite grows from.
Every case has four parts
Input, expected behavior, grader, and a saved result. A case missing any part cannot be rerun or audited, which makes it a demo with a spreadsheet rather than evidence.
A metric is a definition plus a procedure, then a number
State what counts as a failure, on which inputs, judged by whom, in writing, before you compute anything. This is the entire difference between a defensible rate and the undefined "severe hallucination rate" that drew a Texas Attorney General settlement in 2024.
The golden set needs three kinds of case on purpose
Incident cases that can never return, adversarial cases you can anticipate, and ordinary cases that reveal whether a fix broke the common path. A suite of failures alone will pass a system that fails everything else.
The regression gate makes your disclosure letter true
A one-hundred-percent threshold on incident cases, wired to block releases, turns "it will not happen again" from a hope into a rule enforced by a machine that does not get tired at 2 a.m.
Set thresholds before you see results, and hold them
A threshold chosen after the fact is a rationalization. If the honest number is below your line, the gap is the finding, not a reason to move the line.
Validate your grader, especially a model-as-judge
A model grading its own family's outputs shares its blind spots. Check the judge against human labels before trusting its number, and report the grader's error alongside the score.
Your suite proves what is in it, never safety
A perfect score means the failures you thought of passed. The next incident is the one you did not think of, so the suite must grow with every incident and red-team finding, and every result carries a date and a distribution.
The suite outlives the incident
It flows into your trust boundary (see Topic 4.4), your evaluation report (see Topic 4.6), and your conformity file (see Topic 5.6). The failure that hurt someone becomes the first permanent test in a growing body of evidence.
The grader has its own error rate
A false negative hides a real failure and inflates your score; a false positive erodes trust in the suite. Validate the grader against a trusted reference and report its error next to the system's score, or the number means less than it appears to.
Match the metric to the harm
A classifier fails asymmetrically (a missed high-risk case costs differently from a false alarm), a generative system fails on groundedness, an agent fails by acting when it should not. Do not import a generic accuracy number that flatters the system while ignoring the failure that matters.
The suite is evidence, not just a tool
Its dated, versioned cases and results are what let you answer a regulator's real question ("was your decision defensible then") with artifacts rather than feelings. Build it as if a fair but hostile examiner will read it, because one eventually will.
The smallest useful suite is one case
Your Module 3 incident, entered today with a rule and a gate, out-governs a competitor with nothing. Perfect coverage is impossible; a small suite that grows with every failure beats a large one written once from imagination.
Listen to it
The same lesson, talked through by two hosts as episode 27 of the podcast.
Real cases
These examples show evaluation done well and done badly across domains and regions, with the lesson stated plainly. Each is a real, documented case; each is owned for deep treatment by the topic noted, and used here only for its evaluation angle.
Example 1: Pieces Technologies and the undefined rate (United States, 2024). A Dallas healthcare AI vendor claimed a "severe hallucination rate" below one in one hundred thousand for its clinical summarization product, deployed at several Texas hospitals. The Texas Attorney General investigated whether the accuracy claims were deceptive under the Deceptive Trade Practices Act and settled in 2024, with no fine, the company denying wrongdoing, and an agreement to disclose more clearly how it describes performance. Source: Office of the Texas Attorney General, 2024. The evaluation lesson: a headline rate with no published definition, denominator, or grader is not evidence, and a regulator does not need to prove the model is bad to make the claim a liability. This is the anchor case for the topic.
Example 2: The Epic sepsis model and external validation (United States, 2021). A widely deployed proprietary sepsis-prediction model was found, on external validation, to miss most sepsis cases while firing many false alerts, contradicting the vendor's internally reported performance. Source: Wong et al., JAMA Internal Medicine, 2021. The evaluation lesson: a metric computed by the party selling the tool, on its own chosen data, can look very different when an independent group runs a real eval on real hospital patients. Internal numbers are a starting hypothesis, not a finding. (see Topic 4.6) owns this case for the evaluation report; here it shows why "who ran the eval" is part of the metric.
Example 3: Google Flu Trends and distribution drift (United States, 2013). A model that predicted flu prevalence from search behavior over-predicted flu in one hundred of one hundred and eight weeks after user search behavior drifted away from the fixed patterns it was trained on. Source: Lazer et al., Science, 2014. The evaluation lesson: a suite that scored well when built can silently rot as the input distribution moves, which is why every eval result carries a date and a stated distribution. (see Topic 4.5) owns drift; here it is the reason a static golden set is not forever.
Example 4: A model grading its own homework (general practice). A common and dangerous shortcut is to measure a summarizer's hallucination rate using the same model family that produced the summary as the judge. The judge shares the writer's blind spots and will systematically under-count exactly the errors the writer is prone to. The fix is to check the model-as-judge against human labels on a sample before trusting it, and to prefer an independent grader for high-stakes claims. This is not tied to one incident; it is the single most frequent way a good-looking eval number is quietly false.
Example 5: The FrontierMath funding disclosure (United States and international, 2024 to 2025). A benchmark on which a frontier model scored well was later revealed to have been funded by the model's own maker, with model access to the questions not disclosed at the time. Source: reporting via TechCrunch, 2025. The evaluation lesson: even a "public benchmark" can be compromised by who built it and who saw the questions, which is another reason your own suite, built from your own failures, is the evidence you can actually stand behind. (see Topic 12.2) owns this case; here it reinforces that a number's trustworthiness depends on the independence of the evaluation.
Example 6: A recruiting screen that passed its demo (United States, mid 2010s). A large employer's experimental resume-screening AI was found to systematically down-rank women, because it learned from a historically male applicant pool; it looked fine in demonstrations that did not test for disparate outcomes by group. The organization abandoned it. This case is owned elsewhere in the program spine (see Topic 5.4) for the hiring-law angle, and it is referenced here only to make one evaluation point: a demo selects for success and hides group-level failures that a properly designed eval, with cases stratified by the protected attribute, would surface. An eval catches what a demo is built not to see.
Example 7: A dealership chatbot broken by its own users (United States, 2023). A car dealership put a general-purpose chatbot on its website and users quickly steered it into agreeing to sell a vehicle for one dollar and praising a competitor. Source: reporting via Business Insider, 2023. The evaluation lesson: the failures that matter are often the ones a normal test set never contains, because they come from users behaving adversarially rather than cooperatively. An eval suite that includes only well-formed customer questions scores perfectly while the real exposure sits in the injection cases nobody added. This case is owned by Topic 4.3 for red-teaming; here it motivates the adversarial third of the golden set.
Example 8: Behavior that changed between versions (United States, 2023). Researchers measured the same commercial model family on the same tasks a few months apart and found large behavioral shifts, including a sharp drop in accuracy on one arithmetic task between versions. Source: Chen, Zaharia, and Zou, arXiv, 2023. The evaluation lesson: a suite you passed on version A tells you nothing about version B, because a model update you did not control can move behavior in either direction without warning. This is the mechanical reason the regression gate reruns on every release rather than trusting a prior pass. This case is owned by Topic 10.3 for living model cards; here it is the argument for rerunning, not remembering.
Example 9: An automated system with no case for the failure it caused (United States, 2013 to 2017). A state unemployment system automatically adjudicated fraud and wrongly accused roughly forty thousand people, because no check stood between an ambiguous signal and a life-altering accusation. Source: University of Michigan Ford School MiDAS explainer; Michigan Attorney General settlement materials. The evaluation lesson: the failures that ruin people are frequently the ones no test was written for, in exactly the high-stakes, irreversible situations where a case was most needed. An eval suite for a decision system must weight its incident and adversarial cases toward the outcomes that cannot be undone, not toward the easy ones that are pleasant to pass. This case is owned by Topic 3.2 for build-versus-buy; here it is the reminder that a missing case is a missing safeguard.
Where people go wrong
Most of these are not errors of intelligence; they are errors of vocabulary and habit. A capable team makes them because the shortcut feels reasonable in the moment and the cost arrives later, often on a customer's screen. Reading them as things other people do misses the point. Read them as the specific ways your own next eval could quietly fail, and notice how many trace back to the same root: a number that was never made reproducible.
- "We tested it thoroughly before launch." Almost always this means the team demoed it thoroughly. A demo selects inputs that succeed. Your Module 3 incident passed the demo, because nobody demos the failure they have not imagined yet. Testing that counts has fixed inputs, expected behavior, and a reproducible count. If you cannot rerun it next month and compare, it was a demo.
- "Our unit tests pass, so the model works." Unit tests check the code, not the judgment. The retrieval function can return the right document and the prompt can fill in the right fields while the model still writes a summary that invents a drug. Code correctness and behavioral correctness are different axes. Your incident lived in the second one, which unit tests do not touch.
- "A public benchmark score proves our system is reliable." A benchmark measures general ranking on someone else's task. It does not contain your users, your domain, or your incident. A model can top a leaderboard and fail the one case that matters to you. Worse, benchmarks can be compromised by who funded them or who saw the questions. (see Topic 12.2) Build your own suite from your own failures.
- "We have a hallucination rate of X percent." A rate with no published definition of failure, no stated denominator, and no named grader is not a measurement. It is the exact shape of the claim that put a real vendor across the table from a state Attorney General in 2024. Never state a rate you cannot decompose into what counts, on which inputs, judged by whom.
- "The model can grade its own outputs." A model-as-judge drawn from the same family as the writer shares the writer's blind spots and under-counts the writer's typical errors. Model-as-judge is a legitimate, scalable tool, and it must be validated against human labels before you trust its number, especially for the failure class you most care about.
- "We set the threshold after we saw the results." A threshold chosen after the fact is a rationalization, not a standard. Decide the pass level before running, tie it to the stakes, and hold it. If the honest result is below your threshold, the gap is the finding, not a reason to move the line.
- "If the suite passes, the system is safe." The suite only tests what is in it. A perfect score means the failures you thought of did not occur. The next incident is, by definition, the one you did not think of. Passing evals is evidence, never proof of safety, and the suite must grow with every incident and red-team finding to stay honest.
- "One good number covers the whole system." Different case classes carry different stakes and deserve different thresholds and different numbers. Collapsing incident cases, adversarial cases, and ordinary cases into one average hides the failures that matter most behind the volume of the ones that do not.
- "The eval we ran last year still holds." Input distributions drift as users and the world change, and a static suite silently ages. (see Topic 4.5) Every result carries a date and a stated distribution, and the suite is rerun on the current version, not trusted from memory.
- "The grader is right by definition." The grader is an instrument with its own error rate. A model-as-judge can wave through a real failure (a false negative that inflates your score) or reject a correct output (a false positive that trains the team to ignore red). A number graded by an unvalidated grader inherits the grader's error and is not the number you think it is. Measure the grader against a trusted reference before you trust it.
- "We only need to test the failures." A suite of only incident and adversarial cases will pass a system that has regressed on every ordinary input, because it never checks the common path. Ordinary cases are the control group that reveals whether a narrow fix caused wide collateral damage. Drop them and you are blind to the most common way a fix makes things worse.
- "We defined the metric, so we have a good evaluation." A tight failure definition is necessary and it is not sufficient. A metric graded on a golden set built only from easy cases, or pulled entirely from one clinic, one time period, or one type of user, produces a clean, reproducible, and still misleading number, because the ambiguity simply moved from the definition into the denominator. Defining the metric well is half the job; building a genuinely representative golden set is the other half, and the two failures look identical on the page until someone checks where the cases actually came from.
- "Running the suite after we ship is fine." A regression gate that runs after release is an incident report, not a gate. The suite must run before the release decision, or it cannot block anything. Post-hoc evaluation tells you what you already broke; pre-release evaluation is what stops you from breaking it.
- "A model that tops the leaderboard is safe for our use." General ranking on a public benchmark does not contain your users, your domain, or your incident, and some benchmarks are compromised by who funded them or who saw the questions. A leaderboard result is a hypothesis about general capability, never evidence about your specific task. (see Topic 12.2)
Questions people ask
- What is evaluation suite (eval suite)?
- A repeatable, documented collection of test cases run against an AI system to measure its behavior. Each case has a fixed input, a stated expected behavior, and a grader, and the suite produces a number that can be reproduced by anyone with the same cases and definitions. Distinct from a demo, a unit test, and a benchmark.
- What is case?
- The atomic unit of an eval suite, consisting of an input (the exact thing fed to the system), an expected behavior (what a correct system should do, often stated as a rule), a grader (the mechanism that decides pass or fail), and a saved result (the actual output, kept for audit).
- What is golden set?
- A fixed collection of inputs with known correct behavior that is rerun on every version of the system. A well-built golden set contains incident cases, adversarial cases, and ordinary cases in a mix justified by the stakes of the task.
- What is incident case?
- A case built from a failure the system actually produced, entered verbatim, and kept permanently so the same failure cannot silently return. The incident cases are the growing spine of an eval suite and the content of the regression gate.
- What is adversarial case?
- A case built from an input a hostile or careless user could produce, or a variation that probes a known weakness from a new angle, added before such an input appears in production. Adversarial cases are generated by red-teaming the system. (see Topic 4.3)
Keep going
This lesson builds AI evaluation and testing design, and that page shows the roles that hire for it. Every Certified AI Governance Professional (CAIGP) lesson.
Watch the lesson and prove it
Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.
Watch the lesson and prove it →