Incident forensics: reconstructing a failure from logs when memories disagree
The short answer
Reconstruct from records, do not adjudicate memories
When an incident is disputed, the resolution is not the most credible witness but the merged, corroborated log timeline. Sincere people misremember high-stress events through hindsight bias and self-serving recall, so no volume of interviews settles a factual question that the records can settle. Treat every account as a lead to check, never as a finding.
What you will be able to do
- Explain why human memory of an incident is systematically unreliable (hindsight bias, self-serving recall, and the fog of a live crisis) and why that unreliability is normal rather than dishonest, so you know why logs, not accounts, are the spine of a reconstruction.
- Distinguish what a log proves from what a person remembers, and mark every element of your reconstruction as established from records, inferred from records, or unknown.
- Build a defensible incident timeline from timestamped records, correlating independent sources so that no single log carries a load it cannot bear.
- Detect and correct the clock problem: time skew across systems that, uncorrected, makes a true sequence of events look like a false one.
- Corroborate each claim across at least two independent records, and treat the absence of an expected record as itself a finding rather than a blank.
- Preserve the integrity and chain of custody of the evidence so that your reconstruction survives the challenge that it was tampered with or assembled after the fact.
- Produce a forensic reconstruction memo that states what happened, distinguishes fact from inference, assigns a confidence level to each conclusion, and names what the records cannot tell you.
- Separate a model-behavior incident (the model misbehaved on the inputs it saw) from an infrastructure incident (the system around the model failed), using the model's decision log, so you fix the actual cause rather than the suspected one.
- Use witness interviews correctly, as generators of leads to check against records, without contaminating the witnesses or mistaking their agreement for corroboration.
- Judge the ethics of a reconstruction: follow the evidence where it leads even when it implicates a colleague or your own team, and resist the pull to write the story people want rather than the one the records support.
The ideas, one by one
The clock is the trap
The single most dangerous error is trusting a raw timestamp. Clocks drift and disagree, and a few minutes of skew can invert the apparent order of two events and hand you the wrong cause with total confidence. Reconcile every source to one reference clock, in one time standard, before you infer any causal order. Precision is not accuracy.
No load-bearing claim rests on one witness
Every conclusion that assigns cause or blame needs at least two independent records that share no clock and no common failure. This is the Pikesville rule: the reconstruction held because email provenance, phone records, network logs, tool searches, and motive, all from independent sources, converged on one account. Convergence is what an attacker cannot dislodge.
Absence is evidence, once you earn it
A record that would exist if the event had happened, and does not, is a finding, and often a governance defect worth more than any present record. But absence of evidence is only evidence of absence when the log was reliable; establish that the record would have spoken before you read its silence as a fact.
Mark fact, inference, and unknown on every line
Established means a corroborated record shows it; inferred means the records support it but do not state it, with the supporting facts named and alternatives ruled out; unknown means the records do not settle it. Blurring these three turns a reconstruction into a narrative. A clearly marked unknown is what makes your established facts trustworthy.
Chain of custody is what answers "did you cook it?"
Copy the logs, preserve the originals read-only, hash the files, and log your own analysis steps. A reconstruction that a challenger cannot verify was built on unaltered evidence is fragile no matter how right it is. Defensibility also means disclosing your own gaps before an adversary surfaces them.
Follow the evidence even when it points home
The records go where they go, including at your own team, your vendor, or your own prior decision. A finding softened to protect someone is the first thing a hostile reviewer exposes. The discipline that lets your reconstruction credibly exonerate you is the same one that requires it to indict you when the records do.
The reconstruction surfaces the logging you were missing
The gap where the decisive record did not exist is not just an investigative dead end; it is a governance finding. A high-risk system that cannot produce the log needed to reconstruct its own worst failure has a logging defect, and the European Union Artificial Intelligence Act (Regulation (EU) 2024/1689, Article 12) requires high-risk AI systems to keep such records precisely so incidents can be reconstructed. Fixing the gap is how the next incident becomes reconstructable.
Separate the model from the infrastructure around it
A harmful AI output can come from the model misbehaving on unchanged inputs or from the surrounding system feeding it bad inputs, shipping a wrong version, or mishandling a correct output. The decision log settles which, and confusing the two sends the fix to the wrong place. Before blaming the model, check that its inputs and version were what you expected and its outputs were consistent with them.
Interviews generate leads; records generate findings
Do not ignore the people, and do not let their accounts be your conclusions. Run interviews to learn which records to pull, take each account before showing the witness the logs or another version, and never mistake the agreement of witnesses who could have coordinated for independent corroboration. The engineer's memory points at the alerting log; the alerting log settles what the alert did.
The reconstruction is a dossier artifact, not a one-off
It consumes your Module 3 incident record and your Module 10 logging architecture, and it produces the settled account that goes into your evidence annex and the dossier the auditor inspects in the capstone. Kept properly, it means the audit finds a paper trail rather than a scramble. (see Topic 10.6) (see Topic 13.1)
Listen to it
The same lesson, talked through by two hosts as episode 87 of the podcast.
Real cases
These examples show reconstruction from records applied to real events. The Pikesville case is treated first and in depth because it is the anchor for this topic. The others show the same method (independent records, reconciled sequence, corroboration, and the discipline of fact versus inference) recurring across domains. In each, the lesson is the method, not the verdict.
Example 1: Pikesville High School and the reconstruction that overrode the recording (Baltimore County, Maryland, 2024). In January 2024, audio purporting to capture Pikesville High School principal Eric Eiswert making racist and antisemitic remarks spread through the school community and the media. The human situation was a pure standoff: the principal denied ever speaking the words, and a large number of people had heard what sounded exactly like his voice. Baltimore County police, working with media forensics analysts and the Federal Bureau of Investigation, did not try to weigh the principal's denial against the crowd's certainty. They reconstructed. On the audio itself, independent detection methods run by forensic specialists found the signatures of synthetic generation: an unnaturally clean background, missing breaths and natural pauses, a flat unnatural tone, and abrupt cuts where clips had been stitched and background noise added afterward for realism. Multiple methods converged on a high-confidence finding that the audio was AI-generated. On the digital trail, subpoenaed provider records tied the email account that circulated the clip to an internet service associated with a relative of the school's athletic director, Dazhon Darien, and the account's recovery phone number traced to him; school network logs showed he had repeatedly accessed the network across December and January and searched for AI generation tools linked to paid accounts; and a motive appeared in a separate paper trail, an inquiry the principal had opened into possible mishandling of school funds. Police charged Darien, not the principal whose voice had been used, and the case proceeded to prosecution. In April 2025 Darien entered an Alford plea (maintaining innocence while acknowledging the evidence was sufficient to convict) to a charge of disturbing school operations and was sentenced to four months in jail; Maryland had no AI-specific impersonation law at the time, so a deepfake that put a man out of his job and brought death threats on his family was chargeable only as a school-disturbance offense. In 2026 the Maryland General Assembly passed a bill making it a crime to use AI to impersonate a person with intent to harm, a measure its sponsor tied directly to this case (The Baltimore Banner, 2025 and 2026). The reconstruction is a textbook of this topic's method: independent sources that were never coordinated (audio forensics, email provenance, phone records, network logs, motive) all converged on one account, which is exactly why it held where memory and reputation alone could not have, and why the law then moved to meet the evidence the records had made undeniable. The lesson is that when a recording, the seemingly hardest evidence, is itself the thing in dispute, you do not trust it or reject it on impression; you reconstruct its provenance from independent records.
Example 2: The overnight AI outage where three teams remembered three different nights. In a common enterprise pattern, an automated decision system fails overnight, and the next morning the vendor, the operations team, and the on-call engineer give three incompatible accounts of what happened and whose change caused it. No amount of re-interviewing resolves it, because each account is sincere and shaped by self-serving recall and hindsight. The resolution comes from merging the deployment log, the model decision log, the alerting log, and the customer-contact log into one clock-reconciled timeline, at which point the true order of the model push, the first bad decisions, the first complaint, and the first human response becomes visible and settles the question the interviews could not. This is the scenario the Section 7 lab puts you inside, and it is the everyday form the Pikesville method takes in an AI operations context.
Example 3: Aviation accident investigation and the primacy of the recorder over the witness. Air-safety investigation, one of the most mature reconstruction disciplines in the world, is built on exactly this topic's principle. When an aircraft is lost, investigators do not settle the cause by interviewing survivors and weighing their accounts; they recover the flight data recorder and cockpit voice recorder and reconstruct the sequence from the instruments, then use witness accounts only as leads to corroborate against the recorded data. The National Transportation Safety Board and its international counterparts treat the recorded parameters as the spine and human recollection as secondary, for the same reason this topic does: memory of a high-stress event is unreliable, and the instruments are not. The transfer to AI governance is direct. The model decision log is your flight data recorder, and the discipline of trusting the record over the recollection is the same discipline, borrowed from a field that learned it by counting the cost of getting it wrong.
Example 4: Welfare and benefits algorithms reconstructed only after the records were pried loose. In several jurisdictions, the true behavior of an automated scoring system could only be established after investigators obtained the underlying records: the model, its inputs, and its logs. Until those records were in hand, the operators' account of a neutral, fair system stood unchallenged, because there was nothing independent to check it against. Once the records were reconstructed, the actual behavior of the system diverged sharply from the official account. The transferable lesson here is that a reconstruction is only possible when the records exist and can be obtained, which is why access to logs is itself a governance battleground; the deep, sourced treatment of specific welfare-scoring cases and the logging architecture that makes them reconstructable is owned by Module 10. (see Topic 10.2)
Example 5: The financial trading failure reconstructed minute by minute from system logs. When an automated trading system malfunctions and causes large losses in a short window, regulators reconstruct the event from the exchange's and firm's timestamped records, not from the traders' memories of a chaotic morning. The reconstruction establishes, second by second, what orders the system sent and in what sequence, and it is the record, reconciled across the firm's and the venue's clocks, that establishes the account everyone then reasons from. The deep treatment of a full end-to-end regulatory inspection of an automated trading event is owned by the capstone module. (see Topic 13.1) Here it is an illustration that in high-stakes automated failures, the reconstructed log timeline, not the human account, is the primary evidence, and clock reconciliation across venues is a real and decisive part of the work.
Example 6: The model card and the primary source as a reconstruction discipline. A related discipline, treated fully in the next module, is reading the primary source rather than the summary of it. When you reconstruct what a model actually does, you go to the model's own card, logs, and outputs, not to the press description or the vendor's account, for the same reason you go to the flight recorder rather than the survivor's memory: the record is closer to the event than the narrative about it. (see Topic 12.1) The reconstruction habit and the primary-source habit are the same instinct pointed at different objects.
Example 7: The synthetic-media provenance problem and why detection is corroboration, not proof. The Pikesville reconstruction leaned on audio-forensic detection that flagged the clip as synthetic, but detection alone is a single thread, and single threads are exactly what this topic warns against building on. AI-generated media detection is an active, imperfect field: detectors report probabilities, they disagree, and they can be fooled or can misfire on genuine but unusual recordings. This is precisely why the investigation did not rest the case on the audio analysis and instead corroborated it with independent, non-acoustic threads (the email provenance, the phone records, the network logs, the motive). The transferable lesson is that where your evidence is a probabilistic classifier's output, whether a deepfake detector or a model's own confidence score, you treat it as one witness whose reliability you must characterize, and you triangulate it against records that do not share its failure modes. A reconstruction that would collapse if the detector were wrong is a reconstruction resting on one thread. The emerging discipline of content provenance standards, which attach signed, tamper-evident metadata to media at creation so its origin can be checked rather than guessed, is the longer-term answer to the same problem. The Coalition for Content Provenance and Authenticity (C2PA) publishes the leading open technical specification for this, known to end users as Content Credentials, and a growing set of camera, editing, and platform vendors have begun attaching or checking it; where such signed provenance is present, treat it as a corroborating record in its own right, produced by a process independent of the media itself, rather than relying on detection alone.
Example 8: The medical AI alert that everyone remembered differently. A recurring pattern in hospitals deploying AI early-warning systems is a serious patient event followed by conflicting accounts: the clinician remembers no alert, the vendor insists the alert fired, and the nursing record is silent. Reconstruction from the system's own alert-delivery log, the electronic record's access timestamps, and the device logs establishes whether the alert fired, where it was routed, and whether it was seen, often revealing that an alert fired to a screen nobody was watching or was suppressed by an alarm-fatigue setting nobody remembered changing. The pattern mirrors the Section 5 scenario and the Pikesville lesson exactly: the sincere human accounts conflict, and the independent machine records, reconciled and corroborated, settle both what happened and, importantly, exonerate the clinician whose memory was correct that no alert reached them. The deep, sourced treatment of a specific deployed clinical-AI failure is owned elsewhere in the program; here it illustrates that in safety-critical AI, the alert-delivery record is often the decisive, independent witness.
Example 9: The disputed autonomous-vehicle event where the vehicle's own logs settled the sequence. When an automated vehicle is involved in a serious incident, the human accounts (the operator's, bystanders', the company's) conflict and are shaped by shock and stake, exactly as in every other domain here. Investigators reconstruct the seconds before the event from the vehicle's own recorded data: what the perception system detected and when, what the planner decided, what commands went to the actuators, and what a human operator did or did not do, all on a synchronized clock. The reconstruction establishes the sequence that testimony cannot, and it often reframes the human account, showing an operator was disengaged when they remembered being attentive, or that the system classified and then reclassified an object in a way no witness could have perceived. The deep, sourced treatment of a specific automated-driving fatality and the oversight lessons it carries is owned by the capstone module. (see Topic 13.1) Here it is a further illustration that in the highest-stakes automated failures, the machine's own timestamped record is the primary evidence and the reconciled sequence is what everyone else must then reason from.
Where people go wrong
- "Find the honest witness and believe them." The problem in most incidents is not dishonesty; it is that sincere, competent people genuinely misremember high-stress events through hindsight bias and self-serving recall. No interview settles a factual dispute that the records can settle. Treat every account as a lead to check against a record, never as a finding in itself.
- "The timestamps tell me the order, so I know what caused what." A raw timestamp is only as good as the clock that produced it, and clocks drift and disagree across systems. A clock running a few minutes fast can invert the apparent order of two closely spaced events and hand you the wrong cause with total confidence. Reconcile every clock to one reference before you infer any causal order.
- "I only need to reconcile clocks between systems; ordering events within one system is safe as printed." A single machine's clock can drift meaningfully over the course of a long incident, so two events logged hours apart on the very same host are not guaranteed to be in the order their timestamps show either. Reconciliation is not just a cross-system step; it applies to every clock you rely on, including the one you were not thinking to question.
- "One clear log line is proof." One record is one witness, and a witness can be wrong, incomplete, or misread. Every load-bearing conclusion needs at least two independent records that share no source of error. A single clean line feels like proof and is not, especially if it is the only thing carrying a conclusion that assigns blame.
- "Two logs from the same system agreeing means the fact is solid." Records are only corroborating when they are independent. Two lines from the same service share that service's clock, its bugs, and its blind spots, so their agreement can just be the same error printed twice. Ask what error two records share; if they share one, find a third witness that does not.
- "A missing log is just a blank." Absence is evidence when the record would exist if the event had happened. No override entry, in a system that reliably logs overrides, is strong evidence no override occurred, and it is also a finding about the system. But absence of evidence is only evidence of absence when the record was reliable; establish that the log would have spoken before you read its silence as a fact.
- "My job is to produce a confident answer." Your job is to produce a true account of what the evidence establishes, which includes clearly marked inferences and clearly marked unknowns. An honest "the records do not settle this" is a strength that makes your established facts trustworthy; an overstated inference presented as fact is the thread an adversarial reviewer pulls to unravel everything.
- "Fact and inference are basically the same if the inference is strong." They are not, and blurring them is how a reconstruction becomes a narrative. "The push preceded the denials" is established; "the push caused the denials" is an inference that must name its supporting facts and rule out alternatives. Label which is which on every line, because the reviewer will, and if you have not, they will assume you were hiding the difference.
- "I can analyze the original logs directly." Work on copies and preserve the originals read-only, with a hash recorded, or you cannot answer the accusation that you altered the evidence to reach your conclusion. Chain of custody is not bureaucracy; it is the only thing that stands between your reconstruction and the charge that you cooked it.
- "The reconstruction is done when I have a plausible story." A plausible story is the failure mode, not the goal. The reconstruction is done when every load-bearing claim is corroborated by an independent record, every clock is reconciled, every gap is marked unknown, and you have actively looked for and failed to find evidence against your own leading inference. Plausibility is cheap; survivability under attack is the standard.
- "Follow the evidence, unless it points at my own team." The evidence goes where it goes. A reconstruction that softens a finding because it implicates a colleague, your vendor relationship, or your own prior decision is worthless, because its bias is the first thing a hostile reviewer will expose. The discipline that makes the reconstruction credible when it exonerates you is the same discipline that requires it to indict you when the records do.
- "Precision means accuracy." A timestamp to the millisecond, a log that looks authoritative, a confident interface number: none of these is accurate merely because it is precise. False precision launders a guess into a fact. Earn the right to trust a figure by checking its source and its clock before you build on it.
- "If the decisive log does not exist, I should infer what it would have said." When the record that would settle a question was never written or was lost, mark the gap unknown and name the record that would have settled it. Do not fill it with the most plausible story. The missing record is also a governance finding: a system that cannot produce the log needed to reconstruct its own worst failure has a logging defect that the reconstruction exists to surface.
- "The model gave a bad output, so the model is the problem." Not necessarily. A harmful output can come from the model misbehaving on unchanged inputs (a model-behavior incident) or from the surrounding system feeding it different inputs, shipping a wrong version, or acting on its output badly (an infrastructure incident). The decision log settles which, and confusing the two sends you to fix the wrong thing. Check whether the inputs and version were what you expected before you blame the model.
- "I re-ran the model and got a different answer, so someone changed it." Many models are non-deterministic in deployment and produce varying outputs from the same inputs. An answer you cannot reproduce is not proof of tampering or a version change; it may be normal variation. Reconstruct from the output the model actually produced during the incident, recorded then, not from a fresh re-run, which is a lead at best.
- "Interviewing everyone first will orient me before I read the logs." Taking accounts before securing the records is backward and risks contaminating the witnesses: once they have discussed the incident or seen the channel, their independent recollection is gone. Secure and reconcile the records first, then interview to generate leads you check against those records, and never show a witness the logs or another account before taking theirs.
- "A confident, senior person's account should carry more weight." Seniority does not make memory a recording device. A senior engineer is subject to the same hindsight and self-serving recall as anyone, and the deference their confidence commands makes their unchecked account more dangerous, not less, because it discourages the corroboration step. Weight accounts by the records that confirm them, not by the rank of the person giving them.
- "The board wants one name, so the reconstruction should name one." Forcing a single culprit corrupts a reconstruction of a multi-cause failure, converting a chain of fixable process gaps into a scapegoat and tempting the investigation to stop once a plausible name appears. Report the corroborated causal chain and the specific defects that let it happen, framed as fixes; a name does not prevent the next incident and a fixed process does.
- "A settled reconstruction closes the incident." The reconstruction's most durable output is often the logging defect it surfaced, not the answer to this one incident. If you close the incident without fixing the gap that made a record missing, the next incident of the same class will be just as unreconstructable, so the reconstruction is not done until its defects are logged as fixes that feed the file rebuild in Topic 11.6.
- "My reconstruction is objective and bias-free because it is built on logs, not opinions." The records may be objective; your reading of them is not automatic. You can still favor the hypothesis you formed first (confirmation bias), overweight the log that was easiest to pull (availability bias), or anchor on the first version of events you heard before you saw a single record (anchoring). Logs remove the witnesses' bias from the raw evidence; they do not remove the investigator's, and the discipline of triangulation, ruling out alternatives, and attacking your own conclusion in Section 7 exists precisely because the method is not self-correcting without you applying it on purpose.
Questions people ask
- What is incident forensics?
- The discipline of reconstructing what actually happened during a failure from the evidence the systems left behind, rather than from the accounts of the people involved. Its output is a defensible timeline of established facts, marked inferences, and named unknowns.
- What is reconstruction (forensic reconstruction)?
- The artifact this topic produces: a single, clock-reconciled timeline of an incident merged from independent records, in which every load-bearing claim is corroborated, every gap is marked unknown, and a chain-of-custody record makes it verifiable. It consumes the Module 3 incident record and Module 10 logging architecture and feeds the evidence annex and the capstone dossier.
- What is hindsight bias?
- The well-documented tendency to remember an event as more predictable, and one's own perception of it as clearer, than it actually was, once the outcome is known. It is a reason sincere witnesses misremember incidents and a reason the contemporaneous record is trusted over later recollection. More on Hindsight bias
- What is self-serving recall?
- The tendency to remember one's own actions during an incident as more reasonable, timely, and correct than the contemporaneous record shows. Not deliberate dishonesty; a normal property of memory that makes human accounts unreliable as primary evidence.
- What is timeline (merged timeline)?
- One ordered list of every relevant event from every source, sorted by corrected time and tagged by source. The spine of a reconstruction; the instrument that makes cross-system causal links visible by placing events from different systems adjacent on a shared clock.
Keep going
This lesson builds Evidence collection and audit-ready documentation, and that page shows the roles that hire for it. Every Certified AI Governance Professional (CAIGP) lesson.
Watch the lesson and prove it
Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.
Watch the lesson and prove it →