You deploy an agent: scoping, containment, and the kill switch you test before launch
The short answer
An agent is the one system you must be able to stop
A model answers and waits; an agent acts in a loop, on its own, faster than a human can read. That is its value and its danger. Everything in this topic exists because an actor that takes actions must have a proven off switch, or it should not run.
What you will be able to do
- Scope an agent's deployment to the smallest task and the narrowest authority that still does the job, and state in one sentence what the agent may not do.
- Design containment for a specific agent: the walls (permission limits, spend caps, rate limits, allowlists, sandboxing, approval gates for irreversible actions) that bound the blast radius before the agent acts.
- Distinguish a real kill switch (a stop mechanism that lives outside the agent's control and does not depend on the agent's cooperation) from a fake one (a request the agent can ignore, disable, or route around).
- Define the trip conditions for an agent: who may pull the switch, on what signal, how fast, and what the switch does to in-flight actions and side effects already in the world.
- Test the kill switch before launch under realistic conditions, and record the result as an inspectable artifact rather than a claim.
- Justify "do not deploy at all" as a legitimate, defensible outcome when the risks cannot be contained below a threshold you set in advance.
- Connect your kill-switch test record to the wider agent dossier: the permissions you set (see Topic 7.2), the oversight pattern you chose (see Topic 7.3), the audit trail that reconstructs what happened (see Topic 7.4), and the logging and board inspection that come later (see Topic 10.2), (see Topic 13.2).
- Locate the fast, reliable stop you actually control, distinguishing a stop you own (a credential you revoke, a runtime you terminate) from one that runs through a vendor's support process on someone else's timeline.
- Recognize scope creep as a governance risk, and re-run the scope, containment, and stop test whenever an agent gains a capability, raises a cap, or changes the layer it runs on.
The ideas, one by one
Scope narrows the job and the access to the smallest that works
A well-scoped agent has a one-sentence task with every action nameable on one page, and least-privilege authority built up from zero rather than pared down from broad access. You can only wall off and stop a thing whose shape you know, so a narrow scope is the precondition for every other control.
Containment bounds the worst hour before it happens
Scoped credentials, spend caps, rate caps, allowlists, sandboxing, approval gates on irreversible actions, and staged rollout are each cheap before launch and nearly impossible to add during an incident. The test of real containment is a specific, survivable answer to "what is the worst this agent can do in one hour if every judgment is wrong."
A real kill switch does not ask the agent's permission
It acts on the infrastructure beneath the agent (revoke the credential, cut the egress, kill the process, freeze the account), because you cannot rely on a misbehaving agent to obey a stop instruction, and goal-driven agents have been observed resisting shutdown under adversarial test conditions. If the thing you are stopping could refuse or disable your stop, you have no stop.
Trip conditions are half the switch
Name who may pull it with authority pre-granted, the signals that stop it (automatically where humans are too slow), how fast it must act relative to the agent's speed, and what happens to in-flight actions so the stop leaves a state you can account for. A switch without defined trip conditions is a mechanism nobody knows when or how to use.
An untested kill switch is a diagram, not a capability
The only way to know a stop works is to pull it against a running agent, before launch, and measure how fast it stops and what it leaves behind, with a second person able to pull it cold. Knight Capital lost about USD 440 million in 45 minutes for want of a fast, tested stop; the speed of the harm is set by the agent, the speed of the stop is set by whether you rehearsed it.
"Do not deploy" is a legitimate, defensible outcome
If the test shows you cannot stop the agent fast enough, or cannot account for what the stop leaves behind, and the risk cannot be brought below the line you set in advance, refusing to launch is not timidity. It is exactly what sixteen frontier AI companies committed to at the AI Seoul Summit in 2024 for the extreme case: define the threshold in advance, and be willing to not ship.
The stop must be yours, not the vendor's
At least one fast, reliable stop must sit at a layer you control and the agent cannot reach, because a stop that runs through a third party's support process happens on their timeline, not the incident's. Hold an in-house stop (a credential you revoke, a runtime you terminate) and test that one, since it is the stop that will exist at 2 a.m.
Containment bounds harm; monitoring only reports it
Logging and dashboards tell you the agent did the wrong thing, but only scoped credentials, caps, allowlists, sandboxing, gates, and a tested stop limit how bad the wrong thing can get. Detection is necessary because it tells you to act, but it is never a substitute for the walls that make acting actually bound the damage.
The record is the point
Your kill-switch test record is what turns "we can stop it" into an inspectable fact, and it threads into the wider dossier: the permissions and oversight it consumes (see Topic 7.2) (see Topic 7.3), the audit trail and logging it feeds (see Topic 7.4) (see Topic 10.2), and the board inspection that will ask to see it (see Topic 13.2). Keep it honest, including its limitations, because the later audit should find truth, not polish.
Listen to it
The same lesson, talked through by two hosts as episode 56 of the podcast.
Real cases
These examples show scoping, containment, and tested stops in the wild, drawn from varied places, and each one names the lesson it carries. The Seoul commitments are the anchor and are treated in Sections 3F and 5; these are supporting cases.
Example 1: The frontier labs' safety frameworks after Seoul (global). In the months following the Seoul Summit, several signatory companies published or updated the safety frameworks the commitments called for, setting capability thresholds tied to escalating safeguards. Anthropic's Responsible Scaling Policy, for instance, defines capability thresholds that, if a model reaches them, require upgraded safeguards before the model may be deployed or kept in use, and the company reported activating a higher safeguard standard (ASL-3) for relevant models in May 2025. Source: Anthropic, Responsible Scaling Policy. The teaching point is not the specific labels, which change, but the structure: thresholds defined in advance, a rule that says what happens when one is crossed, and a stated willingness to hold a model back. That is the Seoul discipline made operational, and it is the same shape as your kill-switch trip conditions, one level up.
Example 2: The cloud spend cap that saved a startup (widely reported pattern). A common and well-documented failure mode is an automated process or agent that loops on a paid cloud or model API and generates a runaway bill overnight, sometimes tens of thousands of dollars before anyone notices. The teams that survive it are the ones that set hard budget caps and rate limits in advance, so the loop hits a ceiling and stops instead of running free until the invoice arrives. This is containment as an established practice: the cap does not require anyone to be awake. It is a wall you build once that turns an unbounded loss into a bounded, survivable one. The lesson for an agent is direct: a spend cap is not bureaucracy, it is the difference between a scare and a disaster.
Example 3: Automated trading circuit breakers (United States and globally). After the 2010 Flash Crash, in which algorithmic trading drove a near-trillion-dollar intraday swing, exchanges strengthened market-wide circuit breakers: rules that automatically halt trading when prices move beyond set thresholds within a window. Source: SEC and CFTC findings on the events of 6 May 2010 (the crash itself is treated in depth elsewhere in this program (see Topic 13.2)). The circuit breaker is a kill switch with automatic trip conditions, operating at the market level, outside any single algorithm's control. It embodies the two rules this topic insists on: the stop is automatic where humans are too slow, and it acts on the environment rather than asking the misbehaving actors to please stop. Your agent's automatic trip conditions are a circuit breaker scaled to one deployment.
Example 4: The autonomous vehicle safety driver as a contested stop (United States). The 2018 Uber ATG test-vehicle fatality in Tempe, Arizona, is owned elsewhere in this module for the permissioning lesson (see Topic 7.2), but it carries a containment lesson worth naming here in one line: a stop mechanism that depends on a human who is not actually able to intervene in time is not a working stop. A kill switch, like a safety driver, is only real if it can act inside the window in which acting still helps. Design and test yours against the speed of the harm, not against the comfort of having a button.
Example 5: Project Vend, an agent that needed a wall it did not have (research demonstration, 2025). In Anthropic's Project Vend experiment, an AI agent was left to run a small shop and was talked by customers into discounts and giveaways, ending its run well below where it started (Anthropic and Andon Labs, Project Vend, June 2025). This case is treated for its economics elsewhere (see Topic 13.2), but seen through this topic's lens it is a scoping-and-containment story: an agent given broad latitude over pricing and no hard floor on what it could give away had no wall to stop a slow, polite drift into loss. A spend floor or an approval gate on discounts above a threshold, set before launch, would have bounded exactly this. Not every runaway is dramatic. Some are a friendly agent losing you money one reasonable-sounding concession at a time, which is precisely why containment must be set in advance rather than judged in the moment.
Example 6: The MiDAS and Robodebt automated systems (United States and Australia). Two government automated-decision systems, Michigan's MiDAS unemployment system and Australia's Robodebt scheme, caused mass harm in part because there was no fast, effective way to halt them once they were wrong at scale; both are owned by other topics for their core lessons (see Topic 10.2). The one-line transfer to agents: a system that acts on many people and cannot be quickly stopped will convert a design error into a widespread harm before anyone can intervene. The kill switch is not only about a dramatic runaway. It is about the ability to stop any error from compounding across a population while you fix it.
Example 7: The Seoul-to-Paris threshold work (international, 2024 to 2025). The Seoul commitments deliberately deferred the hard part: the signatories agreed to define their specific risk thresholds with input from governments and others ahead of the next summit, the AI Action Summit held in France in early 2025. Source: UK Government, Frontier AI Safety Commitments. The teaching point is that a threshold is not a slogan; it is a defined, observable line, and defining it well is real work that the companies gave themselves months to do. Your agent's trip conditions are the same kind of work at a smaller scale: "stop if it misbehaves" is a slogan, while "stop automatically if the agent attempts any action outside its allowlist" is a threshold you can actually detect and act on. The lesson is to spend the effort turning a vague intention to stop into a specific, observable condition, because only the specific version fires when it matters.
Example 8: Staged rollout as the norm in safety-critical software (aviation and medical, established practice). In regulated domains such as aviation avionics and medical devices, new automated behavior is not switched on for everyone at once; it is introduced through staged deployment, monitoring, and the ability to revert, because the cost of an unanticipated failure at full scale is unacceptable. This is not an AI-specific idea; it is established engineering practice that agent deployment should inherit rather than reinvent. The transfer is direct: run your agent in shadow mode and on a small slice first, watch the evidence, and widen only as it holds, so that a failure you did not foresee reaches a small population you can help before it reaches everyone. Staged rollout is the containment control that costs the least and prevents the most, precisely because it assumes you will be wrong about something and bounds the cost of being wrong.
Where people go wrong
- "We have a stop button, so we can stop the agent." A button that has never been pressed against a running agent is a claim, not a capability. You do not know how long it takes, whether it actually severs the agent's reach, or what it does to actions in flight, until you test it. The button that exists only in the runbook is the button that fails during the incident. Test it before launch, with a second person, and record the result.
- "The kill switch tells the agent to stop." This is the most dangerous single mistake in agent governance. If the agent is misbehaving, you cannot rely on it to obey a stop instruction, and goal-driven agents have been observed resisting shutdown under adversarial test conditions (see Topic 7.1) (see Topic 7.3). A real kill switch acts on the infrastructure beneath the agent (revoke the credential, cut the egress, kill the process, freeze the account), so the agent's cooperation is never in the loop. If the thing you are stopping could refuse or disable your stop, you have no stop.
- "Containment is something we can add later if there's a problem." Every containment control is cheap before launch and nearly impossible to install during an incident. A spend cap set in advance turns a catastrophe into a scare; a spend cap you try to add while the loop is running arrives after the damage. Build the walls first. You cannot pour concrete during the flood.
- "A broad scope is fine because the agent is smart enough to handle it." Capability is not containment. The smarter and more autonomous the agent, the more important a narrow scope becomes, because a capable agent with broad authority can cause harm across a wide surface fast. Scope narrows what can go wrong to a shape you can enumerate, wall off, and stop. "It's smart" is a reason to scope harder, not a reason to relax.
- "Least privilege slows us down; give the agent broad access and we'll monitor it." Broad access plus monitoring means you watch the harm happen across a wide surface instead of preventing it. Least privilege means the harm cannot happen there in the first place. Monitoring is necessary but it is detection, not containment; it tells you the agent did the bad thing, it does not stop the agent from being able to. Start from zero and add back only what a step provably needs.
- "Approval gates on everything keep us safe." Gating every action recreates the rubber-stamp problem from human oversight: a human clicking approve hundreds of times will click approve on the one that matters without seeing it, and you have manufactured a person to blame rather than a safeguard. Gate only the small set of irreversible or high-consequence actions you flagged in scoping. Let the cheap, reversible work run free. Targeted gates work; blanket gates rot into theater.
- "If the test shows we can't stop it fast enough, we launch anyway and fix it later." A stop that is too slow, or that leaves the world in an unknown state, is a failed test, and launching on a failed safety test is how the large losses happen. The defensible responses are to fix the containment and switch, or, if the risk cannot be brought below the line you set in advance, to not deploy. "Do not deploy" is a legitimate, professional outcome, and it is exactly what the Seoul signatories committed to for the extreme case.
- "In-flight actions will sort themselves out when we hit stop." When you pull the switch, some actions are half-done, and a stop that leaves them in an unknown state trades a runaway agent for a mystery. Define, before launch, what the switch does to drafted-but-not-sent, initiated-but-not-settled, and called-but-not-acknowledged actions, and make sure the stop leaves a record of exactly what completed. Being able to account for the wreckage is part of the switch, not an afterthought.
- "The engineer who built it can stop it, so we're covered." A switch only one person knows how to pull is a switch that fails on the night that person is asleep, on leave, or gone. The authority and the procedure to stop must be pre-granted to named roles and rehearsed by more than one of them. Test the switch with a second person pulling it cold from the runbook, because that is the person who will actually be on call when it matters.
- "We can stop the agent by asking the vendor to cut off API access." If your only stop runs through a third party's support process, you have outsourced your kill switch to someone else's timeline, which is almost never fast enough for an incident. Keep at least one fast, reliable stop entirely within your control (a credential you can revoke, a runtime you can terminate) and treat any vendor-side stop as a backup. Test the one you control, because that is the one that will exist at 2 a.m.
- "The agent is behind a chat window, so a human is always in the loop." A human typing to an agent is not the same as a human reviewing each action the agent takes. Once the agent can call tools, move data, or trigger downstream systems from that conversation, it is acting between the human's messages and often faster than the human reads. The presence of a chat interface is not oversight of the agent's actions; the containment and the stop still have to be real.
- "Our agent only reads and summarizes, so it takes no actions and needs no stop." Confirm that before you rely on it. Many agents that begin as read-only acquire action capabilities (send a summary email, file a ticket, update a record) as they are made more useful, and the scope quietly widens without the containment being revisited. A genuinely read-only agent with no ability to affect any system needs less, but the moment it can change anything in the world, it is an acting agent and this whole discipline applies. Re-check the scope whenever the agent gains a capability, and check for the quiet version of this drift too: a tool you granted for reading can sometimes be chained into a write, for example a "search and open ticket" tool that was granted for lookup but also lets the agent file one, or a database connector with a read-only intent that the underlying driver does not actually enforce. Audit what each granted tool can do, not just what you meant it to do.
- "If someone claims a signal that trips the switch, we should always stop first and ask later." Automatic trip conditions are only safe to leave running unattended if the signal itself is trustworthy. A trip that fires on an unauthenticated report, or on a log line anyone with write access could forge, can be triggered by mistake or by an attacker who wants your agent stopped at a convenient moment, turning a safety mechanism into a denial-of-service lever. Authenticate and rate-limit the inputs to your automatic trips the same way you would any other production control, and treat a suspicious flood of trip signals itself as a signal worth investigating, not just a stop worth honoring blindly.
- "We put safety in the kill switch, so scope and containment can be loose." This inverts the layered defense. The kill switch is the last and least preferred layer, a reaction after harm has begun, and a deployment that leans on it while granting broad scope and thin containment is one slow or failed stop away from disaster. Put as much of your safety as possible in the earlier, preventive layers: a capability never granted cannot be misused, and a wall set in advance bounds the harm before the stop ever has to fire. The kill switch must be real and tested precisely because it is the layer you fall back to, not the layer you rely on.
- "A number for the worst hour means we are safe." A bounded worst-hour answer is necessary but not sufficient; the number also has to be one you can survive and correct. "Up to five hundred wrong customer emails per hour" is bounded and still potentially unacceptable. Tighten the walls (lower the rate cap, narrow the allowlist, stage the rollout) until the worst-hour figure is not just finite but tolerable. Containment is producing a number you can live with, not merely producing a number.
Questions people ask
- What is agent?
- An AI system that takes actions in the world toward a goal, typically in a loop and with some autonomy, rather than only producing an output and waiting. Because an agent acts (sends, moves, deletes, calls other systems) faster than a human can review each step, it is the class of system for which a scoped deployment, containment, and a tested kill switch are essential rather than optional. More on Agent
- What is scoping?
- Narrowing an agent's deployment to the smallest task and the narrowest authority that still accomplish the job. Task scope is the one-sentence job with every action nameable; authority scope is least-privilege access to only the tools and systems each step provably needs. Scoping is the precondition for containment and for a working stop, because you can only wall off and halt a system whose shape you know. More on Scoping
- What is least privilege?
- An established security principle that any actor should be granted only the access it needs to perform its assigned task, for only as long as it needs it. Applied to an agent, it means building up authority from zero and adding back only provably-needed capabilities, rather than granting broad access and trying to remember what to remove. More on Least privilege
- What is containment?
- The set of controls that bound how much harm an agent can do while running, regardless of what it decides. Containment includes scoped and revocable credentials, spend caps, rate caps, allowlists, sandboxing, approval gates on irreversible actions, and staged rollout. Its defining test is a specific, survivable answer to "what is the worst this agent can do in one hour if every judgment it makes is wrong." More on Containment
- What is blast radius?
- The extent of harm an agent could cause given its authority and reach. Every capability granted beyond what the task requires enlarges the blast radius; every capability withheld shrinks it. Naming the capabilities you did not grant, as blast radius avoided, is part of a defensible scope. More on Blast radius
Keep going
This lesson builds Privacy by design and privacy engineering, and that page shows the roles that hire for it. Every Certified AI Governance Professional (CAIGP) lesson.
Watch the lesson and prove it
Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.
Watch the lesson and prove it →