Skip to main content

The evaluation report: evidence that your trust levels are earned, not hoped

The short answer

An evaluation report is a decision, not a score

It answers "for which uses do we trust this system, how far, and why is that defensible," with a named signer who owns the consequences. A test result is an input to the report; it is never the report.

What you will be able to do

  • Evaluate whether a given trust level assigned to an AI system is earned by evidence or merely hoped, using the gap between a vendor's marketed performance and independent measured performance as your first test.
  • Assemble an evaluation report that ties every trust claim to a specific piece of evidence, a stated method, the population it was measured on, and a named limit.
  • Distinguish discrimination from calibration, and explain why a single headline score (for example an area-under-the-curve number) can hide a system that is unsafe at the operating point you will actually use.
  • Judge a performance metric against the prevalence of the thing being predicted, and show why positive predictive value collapses when the event is rare even if sensitivity looks acceptable.
  • Tier trust to use: assign different approved uses (full automation, human in the loop, advisory only, not approved) to the same system based on where the evidence is strong and where it is thin.
  • Defend an evaluation report against the four attacks that break weak ones: the wrong population, the hidden operating point, the missing subgroup, and the frozen snapshot.
  • Decide a review-and-revoke trigger for each trust level, so the report expires on evidence rather than on a calendar the organization forgets.
  • Weigh the false-alarm burden of a warning system, using the number needed to evaluate, and explain why a heavy false-alarm load can cause the very missed cases the system was meant to prevent.
  • Critique an evaluation report handed to you by a vendor or a colleague, identifying which of the seven parts is missing and what measurement you must demand before extending any trust.

The ideas, one by one

Trust is earned by evidence on your population at your operating point, or it is only hoped

The Epic sepsis model was trusted for years on a vendor number (AUC 0.76 to 0.83) that described someone else's patients; independent validation on 38,455 real hospitalizations found 0.63 and a 33 percent catch rate. The gap between those numbers is the gap between hoped and earned.

No single metric is evidence

Every metric is honest about one thing and silent about another. Sensitivity ignores false-alarm cost; specificity hides absolute alert burden under rarity; positive predictive value rides on prevalence; AUC averages over thresholds you will never use; calibration is the number almost nobody checks. Evidence is a small, honest set, each quoted with its conditions.

Prevalence changes what your numbers mean

Positive and negative predictive values depend on how common the event is. A strong predictive value on a case-enriched test set evaporates in a general population where the event is rare. Any predictive value without its prevalence is a number floating free of meaning.

Discrimination is not calibration

A model can rank cases well (good AUC) and still produce risk scores that are numerically wrong (poor calibration). Trusting a "40 percent risk" score is trusting a calibration claim that was probably never tested.

Tier trust to use

A system is strong for some uses and weak for others. Grant full automation only where evidence is strong and stakes are bounded; human-in-the-loop where stakes are high; advisory where evidence is thin; not approved where evidence is absent. Writing "not approved" for a use the business wanted is the report's most valuable sentence.

Reports survive four attacks or they fail

The wrong population, the hidden operating point, the missing subgroup, and the frozen snapshot break weak reports. Build the defense in: measure on the right population, report at the operating point, break performance out by the subgroups that matter, and state the date and version the evidence describes.

Name your own limits

A report that finds its own weakest spot removes the hostile reader's best move. Stated limits are strength; a report with no limits is one whose author did not look.

Trust must expire on evidence

Every trust level needs a review-and-revoke trigger: the observable future event that forces re-evaluation. A vendor model update, a population shift, a monitored metric below its floor. Trust that never expires has stopped paying attention.

False alarms are a governed cost, not a footnote

A warning system that catches enough true cases inside a flood of false ones trains its users to ignore it, which is alert fatigue. Report the false-alarm burden and the number needed to evaluate beside the catch rate, because the people living with the system experience both, and a report that hides the alert cost approves a system users will quietly switch off in their heads.

Someone accountable must sign

The person who computes the metrics is not automatically the person who owns the trust decision. Signing commits an accountable owner to defend every trust level, act on every review trigger, and state every limit honestly. An unsigned report, or one signed by whoever was available, is trust with no owner, the condition every case in this module began from.

The evaluation report is not the model card

The model card describes what the system is and what the builder claims; the evaluation report decides how far you, accountably, trust it on your population. The card is an input to the report, never a substitute. When they disagree, your locally measured report governs your deployment. (see Topic 10.3)

The report travels

This artifact feeds the conformity file, constrains agent permissions, and anchors the evidence annex. Build it strong now and you rebuild less when it is attacked later. (see Topic 5.6) (see Topic 7.2) (see Topic 10.6)

Listen to it

The same lesson, talked through by two hosts as episode 31 of the podcast.

Real cases

These examples show evaluation reports done well, done badly, and not done at all, with the reasoning made explicit each time. They are drawn from different sectors and jurisdictions on purpose: the discipline is the same whether the system predicts sepsis, screens tenants, or answers customers.

Example 1: The Epic Sepsis Model and the report that was never written (United States, healthcare). Covered in depth in Section 3E. The governance lesson in one line: hundreds of hospitals held a trust level with no evaluation report behind it, and it took an independent academic team to write the evidence the deploying organizations should have produced themselves. The 2021 external validation (Wong et al., JAMA Internal Medicine) is, in effect, the evaluation report the hospitals owed their patients, produced years late by outsiders. Every deploying hospital had the ability to run this evaluation on its own data and did not.

Example 2: A tenant-screening score and the cost of an unexamined trust level (United States, housing). Tenant-screening algorithms assign risk scores that landlords use to accept or reject applicants. When such a system is trusted to effectively decide who gets housing, the trust level is high (close to full automation) and the harm of a wrong rejection is severe and hard to reverse. A tenant-screening tool faced a legal settlement and a multi-year restriction on scoring certain applicants under fair-housing law. (see Topic 6.1) for the deep treatment of the US state-law patchwork; the point here is narrow: a proper evaluation report would have forced a subgroup breakdown (Attack 3) and tiered the trust level down from automation to advisory for exactly the applicants the aggregate numbers hid. The absence of that report is what let a high trust level sit on evidence that could not support it.

Example 3: A credit-limit algorithm meets a hostile examiner (United States, finance). When public complaints alleged that an algorithm setting credit limits produced disparate outcomes by gender, a financial regulator opened an investigation, and the operator had to defend a decision made by a system whose evaluation evidence was not organized to answer the subgroup question. (see Topic 8.5) owns the hostile-board treatment; for evaluation-report purposes the lesson is that the defense you can mount under examination is only as good as the report you wrote before the examination. An operator with a tiered report and a subgroup breakdown answers the regulator with a document. An operator without one improvises, and improvisation under examination reads as absence of governance.

Example 4: A well-tiered deployment of a customer assistant (illustrative, cross-sector). Consider an organization that deploys a support assistant and does the evaluation properly. It measures on a held-out set of its own real support tickets, at the production operating point. It finds strong performance on simple factual queries, weaker performance on billing disputes, and poor performance on anything involving a legal commitment. It writes a report that grants Level 1 automation to simple factual answers, Level 2 human-in-the-loop to billing responses, and Level 4 not-approved to any output that could bind the company legally, citing the weak evidence there. When a user later tries to manipulate the assistant into making a commitment (see Topic 4.3), the not-approved tier means the workflow never let the assistant commit in the first place. The report did its job before the attack arrived. This is the shape of a report that earns its trust.

Example 5: Reading a vendor benchmark correctly (cross-sector). A vendor presents a benchmark score for a model your team is considering. The disciplined evaluator treats that score as Section 3B requires: a marketing claim about the vendor's data, not evidence about your deployment. The evaluator asks for the operating point, the population, and the subgroup breakdown, and when those are not available, records the vendor claim as unverified and plans an independent evaluation on local data before granting any trust level above advisory. This is not hostility toward vendors; it is the same discipline that would have protected the Epic-model hospitals. The vendor number goes into Part 2 as a claim to be tested, never into Part 4 as evidence.

Example 6: An impact-assessment standard as a report backbone (international). An organization operating across jurisdictions adopts ISO/IEC 42005:2025, the AI system impact-assessment standard, as the skeleton for its evaluation reports, because it needs one format that satisfies several regulators at once. The standard does not do the evaluation for them; it structures the documentation so that population, method, limits, and lifecycle triggers are all present. The lesson: a good report format is portable, and aligning to a recognized standard means the same report can be shown to more than one authority without a rewrite. Framework alignment is a convenience, not a substitute for the measurement itself.

Example 7: A frontier vendor publishing its own limits (cross-sector, established practice). Leading model developers now publish system cards that report not only headline capabilities but also measured failure rates and refusal behaviors, including uncomfortable ones. This matters for evaluation-report practice because it demonstrates the Section 3E discipline at the source: a document is more credible, not less, when it names where the system falls down. When a supplier's system card openly reports a limitation, the disciplined deployer treats that as a gift, a claim they do not have to discover the hard way, and folds it directly into the limits section of their own evaluation report. When a supplier's card is all strengths and no limits, that silence is itself a finding: it tells you the hard measurement was either not done or not disclosed, and your local evaluation must be correspondingly more thorough. Read a card's candor as a signal about the report you now have to write.

Example 8: A public-sector benefits system and evidence that was never designed (Europe, illustrative of a documented pattern). Automated risk-scoring in welfare and benefits administration has repeatedly produced discriminatory outcomes that surfaced only after harm, because the deploying agencies could not produce evidence about how the system performed for the groups it flagged. The deep treatments of specific benefits-scandal cases are owned elsewhere in the program (see Topic 10.4), but the evaluation-report lesson is precise and portable: when the subgroup breakdown (Attack 3) is never done, an agency can run a high-trust automated decision for years with no evidence about its most exposed population, and the missing evidence becomes undiscoverable exactly when it is most needed to defend or correct the system. An evaluation report with a mandatory subgroup section is the artifact that would have forced the question before deployment rather than after a scandal.

Example 9: A hiring-screening tool tiered down before harm (illustrative, cross-sector). A company evaluating an automated resume-screening tool measures it on its own applicant pool, at the production threshold, and breaks the results out by protected groups. The aggregate looks strong, but the subgroup breakdown reveals materially weaker performance for one group. Instead of deploying at full automation as the vendor pitched, the company writes a report that tiers the tool to advisory only for ranking, keeps a human as the decision-maker for every candidate, sets a review trigger tied to quarterly subgroup monitoring, and records the weak subgroup as a named limit with a remediation plan. The tool is used, but never trusted beyond what its evidence earns, and the one measurement that mattered, the subgroup breakdown, is what turned a potential discrimination liability into a governed, defensible deployment. This is the positive mirror image of the benefits-system failure in Example 8: same risk, opposite outcome, and the difference is a report that did the subgroup work before deployment.

Where people go wrong

  • "The vendor's benchmark is our evidence." The most expensive mistake in the module, and the one that cost the Epic-model hospitals years of misplaced trust. A vendor benchmark is a claim measured on the vendor's data, at the vendor's operating point, their way. It goes into your report as a claim to be tested, never as the evidence that earns a trust level. Until you have measured on your own population, you have a hope with a citation, not evidence.
  • "A high AUC means the system is safe to deploy." AUC (area under the curve) summarizes discrimination across all thresholds, but you deploy at one threshold. A model with a respectable AUC can be unsafe at the specific operating point you use. AUC is a screening number that tells you whether any signal exists. It is never, by itself, evidence of safety at your operating point. Always report the operating-point metrics beside it.
  • "Accuracy is a good headline metric." Accuracy (the fraction of all cases the system gets right) is a reasonable headline only when positive and negative cases are roughly balanced; it is nearly useless when the event is rare, because it rewards the model for defaulting to the majority outcome. If sepsis occurs in 6.6 percent of patients, a system that says "no sepsis" to everyone is 93.4 percent accurate and completely worthless. Rare-event evaluation needs sensitivity and positive predictive value together, quoted with the prevalence. Accuracy hides failure in exactly the high-stakes, rare-event settings where evaluation matters most, which is most of the settings this topic cares about.
  • "Positive predictive value is a fixed property of the model." It is not. PPV depends on how common the event is. A model with a strong PPV measured on a case-enriched test set will show a much weaker PPV in a general population where the event is rare. Any PPV quoted without its prevalence is meaningless, and vendors sometimes report PPV on favorable populations for exactly this reason.
  • "One trust level for the whole system." A system is almost never uniformly trustworthy. It is strong for some uses and weak for others. A report that grants a single blanket approval throws away the information a downstream reader needs. Tier the trust: automation where evidence is strong, human-in-the-loop where stakes are high, advisory where evidence is thin, not-approved where evidence is absent. Saying yes and no to the same system is the evaluator's core skill.
  • "Discrimination and calibration are the same thing." They are different. Discrimination (measured by AUC) is the ability to rank cases from lower to higher risk. Calibration is whether the risk numbers mean what they say, so that cases rated 20 percent really do turn positive about 20 percent of the time. A model can rank well and still produce risk scores that are numerically wrong. Calibration is the metric almost nobody asks for, which is exactly where unearned trust hides.
  • "The report is done when the numbers look good." A report with no stated limits and no subgroup breakdown is not finished; it is undefended. The hostile reader's job is to find the weakest population, the hidden operating point, and the missing subgroup. A report that has already named its own weakest spot removes the reader's best moves. Naming your limits is strength, not confession.
  • "Once we write the report, we are done." Trust decays. Populations shift, vendors update models, new failure classes appear in the field. (see Topic 4.5) A report with no review-and-revoke trigger is a snapshot that the organization will treat as permanent long after it stops being true. Every trust level needs a stated condition that forces re-evaluation. Trust that never expires is trust that has stopped paying attention.
  • "Aggregate performance protects everyone." Aggregate numbers are an average, and averages are dominated by the majority. A system that performs well on average can fail severely for a minority subgroup, and the aggregate will hide it. Only a subgroup breakdown makes that failure visible. A report that reports only aggregate metrics has left its most ethically important question unanswered.
  • "A missed case is bad, a false alarm is just annoying." In rare-event, high-stakes settings this ranking is backwards. A flood of false alarms produces alert fatigue, and users who learn to ignore false alarms ignore the true ones too, converting the false-alarm cost into missed cases indirectly. The false-alarm burden and the number needed to evaluate belong in the evidence, weighed alongside the catch rate, not dismissed as a nuisance.
  • "Whoever ran the tests signs the report." The person who computes a metric and the person accountable for trusting the system are different roles, and conflating them is how systems slide into use with no owner. The signer underwrites the trust decision, the review triggers, and the honesty of the limits, and must have the authority and appetite to defend all three. A report signed for convenience is unsigned in every way that matters.
  • "The vendor's model card is our evaluation." A model card is the builder's description of what the system is; an evaluation report is your accountable decision about how far to trust it here. Folding your trust decision into the supplier's card lets the supplier author your governance, which is the population-transfer error in a new disguise. Read the card, test its claims, and write your own signed report.
  • "If the model was validated once, it stays validated." Validation is measured on a specific model version, on a specific population, at a specific time. A vendor update, a population shift, or ordinary drift can expire that validation silently. Without a review-and-revoke trigger, a report becomes a certificate the organization treats as permanent while the system underneath it changes. (see Topic 4.5)
  • "A precise-looking number is a precise number." Reporting a metric to three decimal places, such as "AUC 0.823," implies a level of certainty the underlying sample often cannot support, especially for a subgroup with a small count. This false precision is its own kind of misrepresentation: it is not fabricated, but it presents a rough estimate as an exact one. State the sample size behind a number and, where the count is small, say so in plain words rather than letting a clean-looking decimal do the reassuring for you.

Questions people ask

What is evaluation report?
A signed decision document that states, for each use of an AI system, the trust level granted, the evidence that earns it, the method and population behind that evidence, the report's own limits, and the future observation that would force re-evaluation. It is distinct from a test result (an input), a dashboard (ongoing monitoring), and a pass or fail stamp (which discards the information a reader needs).
What is earned trust?
Trust in a system that points to evidence measured on the deployment population, at the operating point actually used, showing the system performs well enough for the specific decision it is trusted to make. Contrasted with hoped trust.
What is hoped trust?
Trust extended to a system on the basis of a vendor claim, a smooth demo, general reputation, or pilot performance on easy cases, without evidence measured on the actual deployment population. The Epic sepsis case is the canonical example: years of trust resting on a benchmark measured on other patients.
What is trust level (tiered)?
The degree of autonomy granted to a system for a specific use, chosen from four tiers: full automation (acts without review), human-in-the-loop (proposes, a human approves), advisory only (an ignorable input, labeled unverified), and not approved (not used for that purpose). Different uses of the same system can and usually should receive different tiers.
What is operating point?
The specific threshold or setting at which a system runs in production. Performance at one operating point tells you nothing reliable about performance at another, which is why a report must state the operating point and quote metrics at it, not only threshold-independent summaries.

Keep going

Watch the lesson and prove it

Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.

Watch the lesson and prove it →