Skip to main content

Continuity When the Model Goes Down

The short answer

Every AI-dependent process has a default continuity plan already, whether anyone wrote it down or not

If nobody classified the process and designed a degraded mode, the default is silence, discovered by whoever is on shift when the outage happens. This topic replaces that default with a deliberate one.

What you will be able to do

  • Classify every AI-dependent process in your organization into one of three continuity classes: pausable, human-fallback, or cannot-stop, using a repeatable test rather than intuition.
  • Design a degraded mode for each class, stating in plain language what the process does the moment the model becomes unavailable, and who or what performs that function instead.
  • Set a recovery objective for each process that the business, not the technical team alone, has explicitly signed, naming the maximum tolerable outage window and the maximum acceptable staleness of the last AI output the process relied on.
  • Analyze a real, documented cloud-provider outage (the Microsoft Azure Front Door incident of 30 July 2024) for the specific mechanism that turned a contained attack into an eight-hour, cross-service disruption, and connect that mechanism to a defense your own continuity plan must not repeat.
  • Schedule a failover test on a fixed cadence, and distinguish a genuine test (the fallback is actually exercised) from a paper test (the fallback is described but never exercised).
  • Build a continuity table for five AI-dependent processes, each row carrying a classification, a degraded mode, a recovery objective, and a last-test date, defensible to a chief risk officer or a hostile board member line by line.
  • Evaluate a vendor's incident-notification and model-change clauses (see Topic 8.7) against your own continuity table, and state exactly where the contract's promise ends and your own plan has to begin.
  • Distinguish a single point of failure that lives inside your own organization from one that lives inside a shared upstream dependency (a cloud region, a content delivery network, a single model provider), and explain why the second kind cannot be engineered away, only planned around.
  • Prioritize a limited continuity budget across five processes, defending which two or three earn a hot, tested failover and which are defensible with a documented pause and an honest customer message instead.
  • Detect partial degradation, not only total outage, by naming the specific monitored signal and threshold that should trigger a degraded mode before the failure becomes undeniable to everyone downstream.
  • Compare a rollback (undoing the organization's own recent release) against a degraded mode (absorbing a vendor's outage) (see Topic 3.4), and explain why an organization needs both rather than treating either as a substitute for the other.

The ideas, one by one

Classify on the harm of a gap, not on the process's importance

A strategically important process can still be pausable if nothing time-critical happens while it waits; a low-profile process can still be cannot-stop if a short gap produces an irreversible harm. The two judgments are not the same question.

A degraded mode is a specific procedure with a named owner, never a vague intention

"The team will handle it" is not a plan. A colleague unfamiliar with the process should be able to execute the degraded mode from the written description alone.

Recovery objectives belong to the business, not the technical team alone

A number invented without the business owner's sign-off will not survive the first real incident; someone with more authority will overrule it on the spot, exactly when the plan needed to hold.

A described fallback and a tested fallback are different things

The Azure Front Door outage happened because a defense mechanism behaved differently under real hostile load than it was assumed to; only a genuine, triggered test closes that same gap in a continuity plan.

Different vendor names can still share one underlying point of failure

Check the infrastructure layer beneath each AI-dependent process, not only the vendor-facing product name, or a table that looks diversified can still fail as a single point of failure.

Test cadence should match continuity class, not a single organization-wide calendar

A cannot-stop process deserves frequent, disciplined testing; a pausable process does not need the same intensity, and treating both alike wastes the effort that the highest-stakes row actually needs.

A stale cached answer can be more dangerous than an honest gap

Set a maximum acceptable staleness for every process that relies on a cached or last-known-good output, and fail safe once it is exceeded, rather than serving an aging answer with no expiry.

The vendor's contract and your continuity table protect against different failures

A strong incident-notification clause tells you sooner that something is wrong (see Topic 8.7); it does not, by itself, tell your organization what to do about it. Both are required, and neither substitutes for the other.

A continuity table goes stale the moment the processes it covers change

Reopen it at contract renewal, at the first material change to any of its five processes, or on a fixed calendar, whichever comes first, the same discipline that keeps a model-change-notice clause meaningful over time.

Spend a limited continuity budget on consequence, not on visibility

The row where a gap cannot be undone earns the fullest, best-tested fallback, even if it is the least visible to a customer; the row a customer would notice first is not automatically the row that deserves the most protection.

A rollback and a degraded mode solve different problems

A rollback undoes your own organization's recent change (see Topic 3.4); a degraded mode absorbs a dependency you do not control going dark. An organization needs both, and neither is a substitute for the other.

Listen to it

The same lesson, talked through by two hosts as episode 67 of the podcast.

Real cases

Example 1 (the anchor): the Microsoft Azure Front Door outage, 30 July 2024. A distributed denial-of-service attack against Azure Front Door and Azure Content Delivery Network produced an unexpected usage spike; Microsoft's own account states plainly that an error in the implementation of its defenses amplified the impact of the attack rather than mitigating it. The resulting disruption ran roughly eight hours, from 11:45 to 19:43 UTC, and degraded or broke the Azure Portal along with a range of Microsoft 365 and Purview services. Read backward: the failure was not the attack, which cloud providers face constantly and generally absorb; it was a defense mechanism behaving differently under real, hostile load than it was assumed to behave, precisely the gap a genuinely exercised failover test, not a described one, is built to catch. (Sources: Microsoft Azure status history and post-incident summary, 30 to 31 July 2024; Help Net Security, "Microsoft: DDoS defense error amplified attack on Azure, leading to outage," 31 July 2024; Reuters and Dark Reading contemporaneous coverage.)

Example 2 (supporting, pointer): insurers excluding AI liabilities. The deep treatment of the shrinking insurance market for AI-related loss belongs to Topic 8.4 (see Topic 8.4). This topic's addition: an uncovered continuity gap is not a hypothetical exposure once several major insurers have begun filing to exclude AI-chatbot and agent liabilities from standard policy language; a process with a weak or absent degraded mode is, in effect, a self-insured tail the organization is carrying without having decided to, the same framing Topic 8.4 applies to a thin liability cap.

Example 3 (supporting, pointer): the NHS Federated Data Platform's exit and hosting terms. The deep treatment of this contract's clauses belongs to Topic 8.7 (see Topic 8.7). This topic's addition: exit-management provisions and hosting-location restrictions negotiated at signing are a form of long-horizon continuity planning, protecting what an organization can still do if a vendor relationship ends badly, which is the same discipline this topic applies to a much shorter horizon, what an organization can still do in the hours a vendor is simply unavailable.

Example 4 (illustrative pattern, not a single named incident): the shared-dependency surprise. A recurring, documented pattern across cloud-outage postmortems industry-wide (used here as an illustrative structure grounded in ordinary cloud-architecture practice, not a single named AI case): an organization believes it has diversified risk across several vendor-branded tools, discovers during an outage that every one of them sits on the same underlying cloud region or content delivery network, and experiences a full, simultaneous outage of tools it had assumed were independent. Read backward: a continuity table built only at the vendor-name layer, without checking the infrastructure layer beneath it as Section 3F describes, can look diversified and still fail as a single point of failure the first time that shared layer goes down.

Example 5 (supporting, pointer): the two a.m. incident and the value of a plan already written. The deep treatment of running a live AI incident response belongs to Topic 3.5 (see Topic 3.5). This topic's addition: every improvised decision that a responder has to make live during that two a.m. call (whether to pause, who takes over manually, how stale the last output can be trusted) is a decision a completed continuity table has already made in advance, in a calm room, with the business owner's agreement, turning the incident response from an invention exercise into an execution of a known plan.

Example 6 (illustrative pattern, not a single named incident): the tested fallback that was never actually tested. A pattern documented repeatedly across operational-resilience postmortems in banking, telecommunications, and cloud services (used here as an illustrative structure, not a single named AI case): an organization's continuity documentation describes a fallback in confident, specific language, a named team, a named procedure, a named recovery time, and the fallback is discovered, only during a real outage, to have quietly stopped working months earlier (a phone tree with disconnected numbers, a script referencing a system that was decommissioned, a team that was reorganized without anyone updating the plan). Read backward: a written degraded mode is a claim, not a fact, until a genuine, scheduled test exercises it and confirms the claim still holds.

Example 7 (supporting, pointer): model drift as a quieter cousin of an outage. The deep treatment of quiet model degradation belongs to Topic 4.5 (see Topic 4.5). This topic's addition: the maximum-acceptable-staleness discipline built here for a cached fallback output applies equally to a model that never technically went down but has quietly drifted past the point its outputs should be trusted without re-validation, showing that a sudden outage and a slow drift are two different causes converging on the same governance answer, a defined expiry on trust.

Where people go wrong

  • "Continuity planning is only for cannot-stop processes." Every process needs a named degraded mode, including the pausable ones. A pausable process with no queue-and-message design does not fail gracefully; it fails silently, which is a worse customer experience than a clearly communicated pause, even though the underlying harm is small.
  • "Importance and continuity class are the same judgment." A strategically important process (a quarterly planning assistant leadership relies on) can still be pausable, because nothing time-critical happens while it waits. Classify on the harm of a gap, not on how much the business values the process day to day.
  • "A described fallback is a tested fallback." A document stating what should happen during an outage proves nothing about what will happen. Only a genuine, scheduled exercise of the degraded mode, with the fallback actually triggered and measured, closes the gap between the plan on paper and the plan's real behavior, the same gap the Azure Front Door defense mechanism fell into.
  • "The technical team can set the recovery objective alone." A recovery number invented without the business owner's sign-off will not survive the first real incident; someone with more authority will overrule it on the spot, and the plan will have been worthless exactly when it mattered.
  • "Different AI vendor names mean diversified risk." Two processes running on different AI products can still share the same underlying cloud region or content delivery network. A continuity table that checks only the vendor layer, not the infrastructure layer beneath it, can look diversified and still be a single point of failure.
  • "A human fallback works at any volume." A human-fallback plan sized only for normal volume can collapse exactly when it is needed most, since an outage often coincides with a surge in the very demand the fallback exists to absorb. Name the volume threshold honestly and name what happens beyond it.
  • "A cached last-known-good answer is always safer than no answer." A cached decision with no staleness expiry can quietly become a confidently wrong answer rather than an honest gap. Set the maximum acceptable staleness in advance and fail safe (deny, escalate, halt) once it is exceeded, rather than serving an aging cache indefinitely.
  • "Testing every process on the same schedule is thorough." It is actually a misallocation. A uniform test calendar spends the same effort on a low-stakes pausable row as on a cannot-stop row, and creates a false sense that both have been tested with equal rigor when only one of them needed to be.
  • "A continuity table, once built, does not need to change." A table built for the AI-dependent processes an organization runs today goes stale the moment those processes change, exactly the way a model-change-notice clause protects against a vendor's model changing underneath the contract (see Topic 8.7). Reopen the table at renewal, or the first time any of its five processes materially changes, whichever comes first.
  • "The vendor's incident-notification clause is the continuity plan." A strong notification clause tells the organization sooner that a problem exists; it does not, by itself, tell anyone what to do about it. The clause and the continuity table answer two different questions, and winning the first does not excuse skipping the second.
  • "Building the redundant path for a cannot-stop process is a one-time engineering cost." A redundant technical path that is built once and never exercised again drifts out of sync with the primary path it is meant to replace, as data formats change, as thresholds are retuned, or as the primary system's behavior evolves. The fallback needs the same ongoing maintenance and the same scheduled testing as the primary path, not a single build-and-forget project.
  • "A continuity table is a compliance document, not an operational one." A table built only to satisfy an audit checklist, without a genuine intention that a colleague could execute it during a real outage, fails at the one moment it exists for. Write the table to be used at two in the morning by someone who has never seen it before, not merely to be filed after a review.
  • "Once the outage ends, the degraded mode's job is done." Whatever the degraded mode produced while it was active, a queue of paused requests, a batch of manually routed claims, a set of conservatively denied transactions, still has to be reconciled against normal operation once the primary path returns. A continuity plan that ends at "the model is back" without naming how the transition back is handled leaves a gap of its own, the reconciliation gap named in Section 3G's fifth edge case.

Questions people ask

What is continuity table?
The artifact this topic produces: a row per AI-dependent process, each carrying a continuity class, a degraded mode, two recovery numbers, and a last genuine test date, defensible to a hostile board member line by line.
What is continuity class?
One of three categories (Pausable, Human-fallback, Cannot-stop) a process is sorted into based on the shortest time before someone outside the organization is harmed, misled, or meaningfully inconvenienced by the model's silence.
What is pausable process?
A process where nothing time-critical happens while the model is unavailable; the process simply queues or waits, with a visible message, and resumes when the model returns.
What is human-fallback process?
A process where a trained person can perform the model's function manually, at a slower pace and usually a higher cost, for a bounded window before the fallback team itself becomes overwhelmed.
What is cannot-stop process?
A process where no human substitute exists at the required speed or volume, and any real gap produces an irreversible harm, requiring a redundant technical path rather than a pause or a person.

Keep going

Watch the lesson and prove it

Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.

Watch the lesson and prove it →