Skip to main content

Evidence is designed, not gathered: building systems whose proof exists before anyone asks

The short answer

Evidence is designed, not gathered

For any consequential AI decision, the proof is created at the moment of the decision or it never exists. You cannot collect it later; you can only reconstruct a narrative, and a reconstruction fails under challenge. The work is at design time.

What you will be able to do

  • Distinguish designed evidence (created contemporaneously, by the system, at the moment of a decision) from gathered evidence (reconstructed after the fact), and explain why only the first survives a serious challenge.
  • Analyze a single AI decision into its evidence requirements: what fact must be provable, what record proves it, when that record must be created, and who or what must be attributable in it.
  • Identify the reconstruction gap in a system: the distance between what actually happened and what the organization can prove happened, and locate where in the lifecycle that gap opens.
  • Apply the four tests of admissible evidence (contemporaneous, attributable, tamper-evident, complete) to a record and judge whether it would hold.
  • Explain why the burden of proof sits on the operator of a high-risk AI system, using the accountability principle in data-protection and AI law, so that silence is read against you rather than for you.
  • Rank the records you hold by evidentiary strength, from the system's own contemporaneous log down to after-the-fact narrative, and judge whether your consequential decisions are anchored at the top of that hierarchy.
  • Trace the chain of custody for a record, so that a proof you designed is one you can actually produce, intact, when the demand arrives.
  • Design an evidence plan for one system you run: a decision-by-decision map of what proof must exist before anyone asks for it, feeding the evidence annex you assemble in Topic 10.6.

The ideas, one by one

The burden of proof is on you

Under the accountability principle (GDPR Article 5(2)) and the EU AI Act's demonstration and record-keeping duties, an operator of a high-risk system must be able to demonstrate compliance. Absence of evidence is read against you, so silence is a loss, not a draw.

Four tests decide whether a record holds

It must be contemporaneous (created at the decision), attributable (names the model version, inputs, and human actor), tamper-evident (a later change would be detectable), and complete (covers the decisions that matter). A failure on contemporaneity can never be fixed afterward.

The reconstruction gap is your real exposure

Every system has a gap between what it did and what it can prove it did. The gap is silent until someone probes it, and then it is too late to close. Analyzing and closing that gap, decision by decision, is the whole job.

Most things that feel like evidence are not

A policy proves you wrote a policy; an approval proves someone approved; an aggregate metric describes a population; a screenshot was made after the fact. Evidence is a contemporaneous, attributable record of the specific decision under challenge.

The evidence plan is the specification for the whole module

The decision-by-decision plan you built here tells the logging (see Topic 10.2), the cards (see Topic 10.3), the joint impact assessment (see Topic 10.4), and the evidence annex (see Topic 10.6) exactly what to capture, so the file you assemble is a paper trail and not a scramble.

Not all evidence is equal, and a record you cannot produce is not evidence

Anchor your consequential decisions on the strongest layer, the system's own contemporaneous record, not on memory or narrative, and make sure the chain of custody lets you actually retrieve each record intact when it is demanded. A perfect record you cannot lay your hands on closes no gap.

The cost is asymmetric, so postponing is not saving

Designing a record in is a small, one-time task; creating it after the fact is impossible at any price, because the moment is gone. Declining to design evidence in does not save the cost, it converts a small certain cost now into an unbounded exposure the day someone asks.

Designed evidence is due process, not just defense

Records that let you defend a decision are the same records that let the affected person contest it. Designing evidence in is how you make your AI decisions accountable to the people who live with them, which is why the toeslagenaffaire's missing records were a denial of justice, not only a legal exposure.

Listen to it

The same lesson, talked through by two hosts as episode 76 of the podcast.

Real cases

These are real, sourced cases of what happens when evidence is designed in versus left to be gathered. The anchor is examined in depth; the others are drawn from different fields to keep the lesson globally grounded.

Anchor, examined: the Dutch childcare benefits scandal (toeslagenaffaire), Netherlands, exposed 2018 to 2021. The Belastingdienst ran a risk-classification model over childcare-benefit claims from roughly 2013 to 2019, flagging claims on signals that included holding a second nationality and frequent provider changes. Around 26,000 families were wrongly branded as fraudsters and ordered to repay benefits in full, driving many into severe hardship; the affair contributed to the resignation of the third Rutte cabinet on 15 January 2021 (Amnesty International, "Xenophobic Machines: Discrimination through unregulated use of algorithms in the Dutch childcare benefits scandal," 2021). The evidence dimension is the one this topic teaches. When families and investigators demanded to know why individuals had been flagged and on what basis repayment was ordered, the records were incomplete, dispersed, and in places absent; the Dutch data protection authority (Autoriteit Persoonsgegevens) found unlawful processing, and the inability to account for what the system had done was part of the failure, not a side issue. Crucially, none of that evidence could be created after the fact. The decisions were years old. This is the purest available demonstration that evidence must be designed in at decision time or it will not exist when it is finally demanded. (Note: the Rotterdam welfare-fraud algorithm, where investigators had to obtain the source code, model, and training data before they could prove discrimination, is the anchor for the logging-architecture topic (see Topic 10.2); France's CNAF benefits risk-scoring is the anchor for the joint impact assessment (see Topic 10.4). Each is a distinct case; they are named here only to point you to their owning topics.)

Designed-in evidence as a discipline: the aviation flight data recorder. Commercial aircraft carry flight data recorders and cockpit voice recorders, crash-protected devices that continuously capture flight parameters and cockpit audio, mandated by aviation authorities worldwide (for example under International Civil Aviation Organization standards). The entire premise is that the evidence needed to understand a catastrophe must already exist at the moment the catastrophe occurs, because it can never be reconstructed afterward. Nobody "gathers" flight data after a crash; it was designed to be recorded before the aircraft ever flew. This is the mature-industry version of the principle: high-consequence systems record their own decisions contemporaneously and tamper-evidently as a condition of operating. AI governance is early in learning what aviation learned decades ago.

Accountability as a legal duty: the GDPR accountability principle. The General Data Protection Regulation (Regulation (EU) 2016/679, applicable since 2018) codifies in Article 5(2) that a controller must not only comply with the data-protection principles but "be able to demonstrate compliance." This is designed-evidence thinking written into law: it is not enough to have done the right thing, you must have made the doing provable. Organizations that internalized this built records-of-processing and decision logs as a matter of course; organizations that treated it as paperwork discovered at their first serious complaint that demonstrability is a capability you either built in advance or do not have.

The management-system version: ISO/IEC 42001:2023. The first certifiable AI management-system standard (ISO/IEC 42001:2023, published December 2023) is built around documented evidence: to be certified, an organization must show not just that it has AI governance processes but that those processes produce records an external auditor can inspect. The standard operationalizes the same idea at the organizational level: a management system that cannot produce contemporaneous evidence of its own operation is not certifiable, because from an auditor's chair an undocumented control and an absent control are indistinguishable.

Contemporaneous records as a legal standard: business-records rules. Long before AI, evidence law developed the concept that a record made in the regular course of business, at or near the time of the event, by someone with knowledge, is treated as more reliable than an account assembled later for litigation (the business-records exception to hearsay, reflected in rules such as the US Federal Rules of Evidence 803(6) and analogous provisions in many jurisdictions). The law encodes exactly the distinction this topic teaches: a record made routinely at the time carries weight, and a record made after the dispute began, by an interested party, does not. When you design contemporaneous decision records, you are building evidence that fits the shape the law already trusts; when you reconstruct, you are producing precisely the after-the-fact account the rules were written to discount.

The through-line across all five is one lesson, learned repeatedly across a century of high-consequence practice. In every mature domain, and in the law of evidence itself, proof is a designed property of the system, created routinely before it is needed. The alternative, gathering it after the demand, produces exactly the holes and the suspicion that brought a government down. AI governance is simply the newest field learning a discipline that aviation, medicine, finance, and the courts settled long ago.

Where people go wrong

  • "We can gather the evidence when someone asks for it." This is the mistake the whole topic exists to kill. For the decisions that matter, the evidence is contemporaneous or it is nothing: if it was not created at the moment of the decision, it cannot be created later, only reconstructed, and a reconstruction fails the tests that make evidence hold. Asking is exactly when it is too late.
  • "We log everything, so we are covered." Volume is not coverage. Systems routinely log every technical event and miss the consequential human ones, the override, the reason, the person who signed off, which is precisely where a challenge lands. Completeness is defined by the decisions a challenger will care about, not by the gigabytes you happen to store. Undirected logging produces a haystack with no needle.
  • "Our policy proves we govern well." A policy proves you wrote a policy. Evidence is the contemporaneous record that a specific decision followed the policy. A shelf of immaculate policies with no decision records is one of the weakest audit positions there is, because it demonstrates intent and nothing about conduct.
  • "If there is no evidence either way, we are fine." The reverse. Under the accountability principle (GDPR Article 5(2)) and the EU AI Act's demonstration duties, the burden to prove compliance sits on you. Absence of evidence is read against the operator, not for it. Silence is a loss, not a draw.
  • "A screenshot or an export is our record." A screenshot made after the demand fails contemporaneity and tamper-evidence at once. It can orient an investigator but it is not the evidence. The evidence is the system's own record, created at decision time, stored so it cannot be quietly changed.
  • "The vendor keeps the logs, so the evidence exists." If you cannot retrieve it and put it on the table yourself, you do not have it. Evidence trapped inside a supplier's black box you cannot access is a gap you must close in the contract, before deployment, not a comfort at audit time (see Topic 3.3).
  • "A deployment approval proves the decision was handled correctly." An approval proves someone authorized the system; it says nothing about how the system behaved in the specific case that harmed a person. Authorization is an input to the decision, not a record of it, so offering an approval memo when a challenger asks about conduct is answering a different question. The individual decision needs its own contemporaneous record.
  • "If we have the record, we can produce it." Not if the chain of custody is broken. A record trapped inside a vendor system you cannot access, or overwritten by a later process, or aged out by an undocumented retention rule, is at demand time indistinguishable from a record you never made. Naming the record is only half the work; being able to retrieve it intact is the other half.
  • "Evidence is a compliance cost with no other value." Designed evidence is also what makes your decisions contestable and your good conduct provable. It is the raw material of due process for the people your system affects and the only way to convert "we did the right thing" into "we can show we did the right thing," which are very different positions when someone does not trust you.
  • "More evidence is always better." Evidence should be designed, which means proportionate to consequence. Recording irrelevant detail on trivial decisions buries the records that matter and creates its own privacy and retention risks. The skill is deciding what each consequential decision must prove, not maximizing capture. Design, not accumulation.

Questions people ask

What is evidence (in governance)?
A contemporaneous, attributable, tamper-evident, and complete record of a specific decision, produced by the system that made it. The thing you can put on the table to demonstrate what happened, as distinct from intention, policy, or narrative.
What is designed evidence?
Proof that is specified before a system operates and created automatically, by the system, at the moment of each consequential decision, so that it already exists when anyone asks. The opposite of gathered evidence and the central idea of this topic.
What is gathered evidence?
Proof that an organization tries to assemble after a demand arrives, by digging through memory, exports, and documents. For consequential decisions it is typically incomplete and suspect, because the records were never created contemporaneously and the assembly happens after the outcome is known.
What is reconstruction gap?
The distance between what a system actually did and what the organization can prove it did. It is silent until challenged, cannot be closed retroactively for decisions already made, and is the real measure of an organization's audit exposure.
What is contemporaneous?
Created at the moment of the decision, not reconstructed afterward. The one property of a record that can never be added later, which is why evidence work is front-loaded into design time.

Keep going

Watch the lesson and prove it

Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.

Watch the lesson and prove it →