Skip to main content

The 2 a.m. incident: your system fails live, stakeholders are calling, run the response

The short answer

An incident is a role you step into, not a mood you fall into

At a defined moment (the declaration), a small set of jobs is picked up on purpose. Before it, the feature is one engineer's problem; after it, it is a coordinated response with one decision-maker, one voice out, and one record. The technical problem does not change at the declaration; how humans handle it does, and that is what decides the night.

What you will be able to do

  • Recognize an AI incident early from your own instruments, and apply a simple severity scale to decide, without a debate, whether to formally declare an incident and start the response.
  • Build a severity ladder for your own feature in which each tier is defined by impact rather than by who is shouting, and carries its own response clock, its own notification list, and its own decision authority, agreed before any incident happens.
  • Run the response as defined roles rather than a scramble: name the incident commander (the single decision-maker), the communications lead, and the scribe who keeps the log, and state what each one does and does not decide.
  • Sequence the response correctly under pressure: contain the harm first (pull the tested rollback), then diagnose, because diagnosing a live, machine-speed failure before stopping it is how a bad night becomes a catastrophic one.
  • Write an incident log as the incident happens: timestamped facts, decisions, and who made them, in a form that a post-incident review, a regulator, and a later forensic reconstruction can all read.
  • Communicate with stakeholders under pressure using a holding statement that states what you know, what you are doing, and when you will update, without speculating about cause or assigning blame while the facts are still moving.
  • Distinguish the incident commander's job (decide and coordinate) from the responder's job (execute) and from the expert's temptation (debug the interesting problem), and hold the sequence when every instinct says to start fixing.
  • Account for the machine-speed clock: state, for your own feature, how fast harm accrues per minute and therefore how fast the response must move, and design the response so the slowest step is not "who is allowed to decide."

The ideas, one by one

Declare early, with a scale and a bias to declare

Tie each severity rung to a monitoring signal and threshold you set in daylight, so declaring is crossing a pre-drawn line, not summoning nerve. When unsure between two levels, act at the higher one: a false alarm costs a little coordination, a missed alarm compounds every minute. And wire the alarm to wake a specific, empowered human, because Knight's ninety-seven unread emails are what a great alarm sounds like when it reaches no one.

The ladder is what makes the response fast

A severity scale names the fire; a severity ladder also carries, on each tier, the response clock, the notification list, and the decision authority for that tier. Define the tiers by impact, never by who is shouting, and agree the whole thing on a calm afternoon with the people whose authority it assigns. At 2 a.m. nobody should be inventing the escalation path or arguing about whether this is serious. Every clock and every name in it comes from your own systems and your own obligations, because a borrowed ladder encodes somebody else's harm rate and org chart.

Contain before you diagnose

The single hardest discipline of the night, because every instinct pulls the other way. Stop the harm with the tested off switch from Topic 3.4, then diagnose on a stopped system. Knight's responders reached for the cause and, by widely reported accounts, made it worse; the response that stops the harm first and is clever second wins every time.

One commander, one voice, one record

The incident commander decides and coordinates (and is usually not your best engineer, whose attention belongs on the fix). The communications lead is the single channel out and shields the responders. The scribe keeps the timestamped log. A scramble of five voices taking parallel half-actions is slower and more dangerous than a coordinated few.

The holding statement tells the truth without guessing

Three parts, no fourth: what we know (confirmed only), what we are doing, when we will update. No speculation on cause, no blame, no promised fix time, no minimizing, no false closure while the incident is still open. Silence reads as a cover-up; over-talking creates retractions. Honesty about the partial story is what buys the trust to keep working.

Respect the clock

Your feature, not you, sets how fast harm accrues per minute, and that number sets how fast containment must be and how automated your response must be. Where the harm outruns human reaction, the containment must be an automatic circuit breaker on the pre-registered condition, and the human decision must be about restoring, not stopping. The slowest step in your response must never be "who is allowed to decide."

The log is the deliverable of the night

More durable than the fix, because the fix solves this incident and the log improves every incident after it. It is the spine of tomorrow's post-incident review (see Topic 3.6), the honest customer letter (see Topic 3.7), the Module 4 test case (see Topic 4.2), and any later forensic reconstruction (see Topic 11.4). A response with no record cannot be reviewed, defended, or learned from.

The quiet incidents are the hard ones to catch

AI features usually fail without crashing, returning a fast, confident, wrong answer while every uptime signal stays green, so detection depends on the AI-specific instruments from Topic 3.4, not the golden signals. Judge your detection by how your incidents arrive: your own instrument (mature), a human noticing (slower), or a customer or regulator telling you (worst). A shop that hears about its AI failures from customers has built exposure with a lag, not detection.

Run the response in one channel, and recover as a governed restore

Decisions made in side conversations did not happen, because the commander did not see them and the scribe did not log them, and uncoordinated parallel fixes are how a fault spreads. Keep the response in a single place. And when the harm is contained, treat bringing the feature back as its own small staged, watched rollout after the fix is verified, not a fast triumphant switch flip that reintroduces the fault.

Keep the response and the review blameless

Almost every incident is a system failure with a human at the sharp end, and hunting for someone to punish teaches the organization to hide the next one: declare late, shade the log, avoid escalating. Hold the system accountable (what let one slip cause this, and how do we change it?), not an individual up for punishment. Blaming Knight's technician for the missed server would have left every process hole open for the next one.

This response consumes the launch you built and feeds the review that follows

The tested rollback from Topic 3.4 is your containment; the incident log you write here is what Topic 3.6 dissects and Topic 3.7's letter is built from. Run it as a rehearsed procedure, not a scramble, and a bad night becomes an improvement instead of a catastrophe.

Listen to it

The same lesson, talked through by two hosts as episode 21 of the podcast.

Real cases

These examples show live-incident decisions in real systems, with the reasoning stated. The deep anchor is Knight Capital; the others sharpen a specific point and are treated in depth by their owner topics.

Example 1 (the anchor): the Knight Capital 45-minute meltdown. On 1 August 2012, a deployment error left obsolete "Power Peg" code active on one of Knight Capital's eight order-routing servers. When the market opened, that server began firing orders into the market at machine speed; in about forty-five minutes the system took over four million executions in 154 stocks and cost the firm roughly USD 440 million, nearly destroying it (SEC, "In the Matter of Knight Capital Americas LLC," Release No. 70694, 2013). Read against this topic, Knight failed at almost every stage of the response and the failures were governance failures, not intelligence ones. Detection and declaration: the firm's systems sent ninety-seven automated alert emails naming the problem before the market opened, and no accountable human had the job of reading them, so the alarm reached no one in time. Escalation: the SEC found Knight had no documented procedure for engineers to escalate to senior risk management when an algorithm behaved abnormally, so the people watching the losses could not reach the people who could authorize a stop (SEC, "SEC Charges Knight Capital With Violations of Market Access Rule," 2013). Containment: Knight had no automated way to stop erroneous orders and no rehearsed off switch, so the forty-five minutes were spent diagnosing and improvising while the harm ran; by widely reported accounts, an attempt to fix it by removing the new code from the seven correct servers spread the broken logic to all eight and made it worse, the textbook cost of diagnosing before containing. The regulatory outcome sealed the lesson: Knight settled for a USD 12 million penalty for violating the Market Access Rule, and the SEC's findings read as an inventory of the missing response (no written deployment procedures, no adequate pre-deployment testing, no automated erroneous-order check, no escalation path). Every artifact this topic asks you to build is a direct answer to a specific thing Knight did not have.

Example 2 (a fast, clean containment done right): a stock-exchange circuit breaker. Modern equity markets install automated circuit breakers that halt trading when prices move beyond set thresholds in a short window, a control adopted precisely because machine-speed events showed that human reaction time is not a control at machine speed (US Securities and Exchange Commission market-wide circuit-breaker rules, adopted after the 1987 and 2010 market events). The relevant lesson for AI incidents is the design principle from 3G: when the clock runs faster than a human can decide, the containment must be automated and pre-authorized, tripping on the condition rather than waiting for a person to wake, assess, and approve. The circuit breaker is contain-first, made mechanical: it stops the harm on a pre-registered trigger and hands the humans a stopped system to reason about. A fast-clock AI feature needs the same shape, an automatic trip on the pre-registered condition, with the human decision reserved for restoring, not stopping. Where the market's breaker is a blunt, binary halt (trading stops entirely), an AI feature's automated trip is usually better built graduated (drop the harmful segment, fall back to a cached response, tighten an output-validation filter) rather than always taking the whole feature down; the design principle is the same (pre-authorized, condition-triggered, no human in the loop for the stop), the shape of the stop is what you tune to your feature.

Example 3 (why the alarm must reach a person): monitoring that no one watches. Knight's ninety-seven unread emails are a specific case of a general failure: an instrument that fires into a void. The Topic 3.4 monitoring you built is only half a control; the other half is the wiring from 3C that turns a threshold crossing into a woken human with authority and a scale to act. A team that congratulates itself on rich dashboards and alerting, but has no on-call role that is always filled and always empowered to declare, has built Knight's ninety-seven emails with a nicer interface. The instrument that detects and the human who responds are one system, and an incident is where you find out whether you wired them together.

Example 4 (a slower clock, an irreversible output): a deployed model harming quietly over weeks. Not every incident is fast. A model that degrades slowly, or a summarization feature that fabricates occasionally, harms at a slower rate but can run undetected far longer, and the "incident" may be declared only when an outsider notices, as when Apple's notification summaries were caught fabricating news by a broadcaster rather than by Apple's own instruments (see Topic 3.4). The response discipline is the same (declare, contain, assess, diagnose, communicate, record), but the clock changes what "contain" costs: a slow-clock failure gives you time to prefer a lighter, segmented containment, while its irreversible delivered outputs (the messages already sent, the summaries already read) are the harm no rollback recalls, which is why the honest line about irreversibility from Topic 3.4 belongs in the incident log too. Speed and reversibility are the two dials that set how aggressively you contain.

Example 5 (the incident log as a later defense): reconstructing an incident from records. When an incident is later investigated, by a post-incident review, a regulator, or a court, the only thing that survives is what was recorded. In cases where investigators had to reconstruct an algorithmic harm long after the fact, the presence or absence of a contemporaneous, timestamped record decided whether the reconstruction was reading a trail or guessing at memories; the deep treatment of that forensic problem belongs to Module 11 (see Topic 11.4), and the logging architecture that must produce such records to Module 10 (see Topic 10.2). The point here is that the scribe's dull, timestamped log is not busywork during the crisis; it is the single most durable output of the night, the document that turns tomorrow's review into an investigation of facts rather than a contest of recollections.

Example 7 (blame versus system, the Knight technician): where a review should look. It is tempting, reading the Knight case, to land on "a technician missed one of eight servers" and stop there, as if the disaster were one person's slip. That is true in the narrow sense and useless as a lesson. The SEC's findings point the accountability at the system around the technician: Knight had no written deployment procedures, no peer review of the deployment, no adequate testing of the new code in a production-like environment, no automated check to catch and block erroneous orders, and no escalation path to stop them (SEC, Release No. 70694, and SEC charges, 2013). A blameless analysis asks why a single human slip could nearly destroy a firm, and answers that a robust system would have caught the missed server, caught the erroneous orders, and stopped them fast. This is the posture to carry into your own post-incident review (see Topic 3.6): firing the person who made the proximate error removes one human and leaves every process hole open for the next one, while fixing the system that let one slip become a catastrophe is the only change that transfers. Blameless does not mean consequence-free: it means the response and the review are not run as a hunt for someone to punish, which is different from an organization later concluding, through its normal management process and after the system fix is identified, that a specific pattern (a checklist repeatedly ignored, a step skipped on purpose) needs its own remediation. The distinction matters because "blameless" is sometimes heard as "nothing happens to anyone, ever," and that reading fails under real scrutiny; the discipline is to fix the system first and always, and to keep any individual accountability separate from the incident response itself, decided later, calmly, through ordinary management, not improvised at 2 a.m. or folded into the log. The example is here because the response you run tonight is judged, in the end, by whether it made the system safer or just found someone to blame.

Example 6 (the rushed restore that becomes a second incident): re-enabling before the cause is fixed. A common and avoidable pattern is the response that contains well and then recovers badly. The harm is stopped, the team is relieved, and, wanting to prove the outage was brief, someone re-enables the feature within minutes, before the cause is understood or fixed, and the same failure fires again, now with an exhausted team, a skeptical stakeholder, and a second holding statement that is much harder to write than the first. The discipline that prevents this is to treat recovery as its own small governed rollout (the same staged, watched, reversible re-exposure from Topic 3.4), not as flipping the switch back on. It is frequently correct to leave a feature off overnight and restore it in daylight with the fix verified and someone watching, because the cost of a slightly longer outage is small and recoverable while the cost of a second identical incident is trust you may not get back. Speed of restoration is not the score; a restore that reintroduces the failure is not a recovery, it is the next incident with a running start.

Where people go wrong

  • "We will handle it if it happens; we do not need a written response." This is Knight's posture exactly, and Knight had brilliant people. Under pressure, humans default to the wrong instincts (diagnose before contain, talk without deciding, reassure before knowing), and only a pre-written response overrides the defaults. "We will handle it" means "we will decide who is in charge and what stop means while the clock runs," which is the most expensive time to decide anything.
  • "The person who understands the system best should lead the incident." Usually wrong. Your best engineer's attention belongs on the technical problem; the incident commander's attention belongs on the whole board (contain, communicate, coordinate). If your expert is commanding, either the diagnosis or the coordination is being neglected. Put the expert on the fix and someone else in the commander's seat, even if the someone else knows less about the code.
  • "First, figure out what is wrong." The most natural and most dangerous instinct. Diagnosis takes time you do not have while harm runs at machine speed. Contain first (pull the tested off switch), then diagnose on a stopped system. Knight's responders reached for the cause and, by widely reported accounts, made it worse; the discipline that saves the night is stopping the harm before understanding it.
  • "We should not overreact by declaring an incident over an unclear signal." The asymmetry is the whole point. A false alarm costs a little coordination and an apology; a missed alarm compounds every minute. Build a bias to declare: when unsure between two severity levels, act at the higher one. The fear of looking like you overreacted is exactly the fear that cost Knight its first minutes.
  • "Severity is obvious once you are in it." It is not, and the thing that most often decides it is who is shouting. The loudest signal in an incident is usually the most senior person who has just found out, and a response that lets seniority set severity will run its fastest clock for an executive's bruised morning and its slowest for the quiet failure that is actually harming people. Define each tier by impact instead: who is harmed, how fast, how reversibly, how publicly. The shouting is an input to the communications plan, not to the severity call.
  • "We have a severity scale, so we have a ladder." A scale tells you what to call the incident. A ladder also tells you how fast you must move, who has to be told, and who is allowed to make the call, tier by tier. Without those three attached to the rung, you have named the fire and still have to negotiate the response while it burns. Attach the clock, the notification list, and the decision authority to every tier, and get the people whose authority it assigns to agree to it while nothing is on fire.
  • "Our monitoring will catch it, so we are covered." Monitoring that fires into a void is not a control. Knight's systems sent ninety-seven alert emails that no one was assigned to read. An instrument only becomes a control when it wakes a specific, always-filled on-call role that has the authority and the scale to declare and act. The dashboard and the human are one system.
  • "Turning it off is drastic; let us try to fix it live first." Reversed priorities. It is almost always safe to have stopped a healthy feature by mistake; it is rarely safe to have let a harming one run while you were clever. Containment buys back the time diagnosis needs. "Fix it live first" is how a bounded incident becomes an unbounded one.
  • "Say nothing to stakeholders until we have the full story." Silence reads as incompetence or cover-up and lets rumor become the official story. Issue a holding statement immediately: what we know, what we are doing, when we will update. You do not need the full story to be honest about the partial one; you need the discipline not to guess at the parts you do not have.
  • "Tell them we think a bad deploy caused it and we will be back in thirty minutes." Too much, too soon. Never speculate on cause, assign blame, or promise a fix time while facts are moving; each is a public statement you may have to retract, and a retraction turns one incident into two. State only what is confirmed, and let the next update carry what you have since confirmed.
  • "Everyone should jump in and help; more hands, faster fix." A crowd of responders each taking parallel half-actions is slower and more dangerous than a coordinated few, because uncoordinated fixes collide (this is how removing code from seven servers spread a fault to eight). One commander decides, assigns, and holds the picture; extra hands help only when they execute assigned actions, not when they freelance.
  • "We will write up what happened afterward, from memory." Memory of a stressful hour is unreliable and self-serving, and the details that matter (the exact time containment took hold, who decided what) are the first to blur. The log is written as it happens, by the scribe, timestamped, or it is not a record. Afterward-from-memory is how a post-incident review becomes a contest of recollections instead of an investigation of facts.
  • "A rollback stops the incident, so the incident is over when we roll back." Containment stops new harm; it does not undo the harm already delivered (the messages already sent, the orders already fired), and it is not the end of the response. Assessment, diagnosis, honest communication, careful recovery, and the log all follow. Treating containment as the finish line skips the half of the response that prevents the next incident.
  • "Restore the feature fast to show we recovered quickly." A triumphant re-enable that reintroduces the unfixed cause is a second incident with less sympathy. Recovery is its own small governed rollout: revert the cause, restore carefully, watch it. It is often right to leave the feature off overnight and recover in daylight; speed of restoration is not the score, and a rushed restore is how one bad night becomes two.
  • "Our feature is not machine-speed, so we do not need this." The response discipline (declare, contain, assess, diagnose, communicate, record) is the same at every clock speed; only the required speed of containment changes. A slow-clock feature gives you time to prefer a lighter containment, but a slow failure can run undetected far longer, and its irreversible delivered outputs are still harm no rollback recalls. Slow does not mean safe; it means a different shape of the same response.
  • "The incident commander needs to be senior." The incident commander needs to be trained and available, not senior. Seniority helps for the heaviest authorizations (which the escalation path routes separately), but the commander's job is to run the sequence and coordinate, which a trained on-call person of any rank can do, and often better than an executive parachuting in mid-incident. Confusing rank with the commander role slows the response waiting for the important person to wake up.
  • "Whoever is closest to the problem should just handle it in a side chat." Decisions made outside the one coordination channel did not happen: the commander did not see them, the scribe did not log them, and no one can be sure they did not collide with another responder's action. Side-channel fixes are precisely how uncoordinated changes interact badly (seven servers became eight at Knight). Every system-changing action is proposed, decided, and logged in the single channel, or it is freelancing.
  • "Root cause first; we cannot fix it if we do not understand it." You do not fix it in the incident; you contain it in the incident and fix it afterward, on a stopped system. Understanding is required for the fix, not for the containment, and the containment is what stops the harm. Requiring full understanding before stopping is the exact instinct that spent Knight forty-five minutes; the containment usually needs far less understanding than the fix, sometimes none.
  • "A long incident is just a short one that takes longer." A long incident degrades its own responders: the sharp team at hour one is exhausted by hour four and makes worse calls with sloppier records. Long incidents need planned shift handoffs, with the incoming commander reading the log and taking over at a recorded moment, so hour four is as well-run as hour one. A response with no handoff plan fails when its first people burn out.
  • "Someone was clearly at fault; the review should find who." Almost every incident is a system failure with a human at the sharp end, and hunting for a culprit teaches everyone to hide the next incident (declare late, shade the log, avoid escalating), which forfeits the early visibility good response depends on. Hold the system accountable (what let one slip cause this, and how do we change it?), not an individual up for punishment. Blaming the technician who missed a server leaves every process hole open for the next one.
  • "Our AI feature will page us when it breaks, like our servers do." AI features usually fail without crashing: a fast, confident, wrong output returns a success status and uses normal resources, so the uptime pager stays silent while harm accrues. Only AI-specific instruments (output-quality sampling, groundedness, complaint rate) detect it. A team relying on crash-style alerting for an AI feature will learn about its incidents from customers, which is the most expensive place to learn.
  • "The executive on the phone needs an answer and a name now." Pressure from leadership mid-incident pulls you toward the two forbidden moves: guessing the cause and naming a culprit. Neither is available while facts move, and both create commitments you must walk back. Give leadership what is real (the contained status, the honest customer update, and that the restore decision comes in daylight with the facts), decline to speculate or blame, and get back to the response. A commander who will not be rushed into a guess is doing the job.

Questions people ask

What is incident response?
The rehearsed set of roles, decisions, and steps an organization runs when a live system starts causing harm. It is picked up on purpose at a defined moment (the declaration), not fallen into as a scramble, and its purpose is to override the wrong instincts humans default to under pressure. More on Incident response
What is incident (declared)?
A live failure serious enough that the organization formally starts the response, assigns roles, and contains the harm. The declaration is a decision made against a severity scale, and it is the line between "one engineer looks at a problem" and "a coordinated response with a single decision-maker."
What is severity scale?
A short ladder of rungs (for example SEV-3 minor, SEV-2 major, SEV-1 critical) that turns a vague sense of "this is bad" into a category with a defined response, each rung tied to a monitoring signal and threshold set in advance so declaring is crossing a pre-drawn line.
What is severity ladder?
The severity scale with three more things attached to every rung: the response clock that tier commits you to, the notification list that tier triggers, and the role authorized to decide at that tier. Tiers are defined by impact (who is harmed, how fast, how reversibly, how publicly) and never by the seniority of whoever is shouting. Agreed in advance by the people whose authority it assigns, it is what removes the negotiation from the first minutes of a response; every value in it is set from the organization's own systems and obligations.
What is response clock?
The time a given severity tier commits the organization to, from declaration to the actions that tier requires. It is set against how fast the feature can actually harm someone and against any reporting deadline the organization is subject to, and it is checked against the real elapsed times a drill produces, so a clock the team has never hit is corrected rather than published.

Keep going

Watch the lesson and prove it

Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.

Watch the lesson and prove it →