Skip to main content

Permissioning an agent: tools, budgets, and the actions it may never take alone

The short answer

An agent is a model that acts through tools, and its permission set is what it can actually do

Not what you told it to do or hope it will do, but the tools it holds, bounded by the budgets on them, minus the actions you gated or removed. Every other control in this module operates inside the boundary the permission set draws, which is why permissioning is the first control and not an afterthought.

What you will be able to do

  • Define an AI agent as a system that takes actions in the world through tools, and its permission set as the specific list of tools it may call, the budgets that bound how much it may do, and the actions it may never take without a human.
  • Apply the principle of least privilege to an agent: grant the minimum set of tools and scopes it needs for its task, default to denying every capability you have not deliberately granted, and prefer read-only and sandboxed access wherever the task allows.
  • Set enforced budgets that bound an agent's blast radius, including spend caps, rate limits, a step or iteration cap for loops, a time limit, and a scope limit on which accounts or records it may touch.
  • Separate the actions an agent may take alone from the actions it may never take alone, and within the second group, distinguish actions a human must authorize before execution from actions the agent is forbidden entirely.
  • Distinguish an enforced permission, a limit the agent cannot exceed because the tool or harness refuses, from an instructed permission, a sentence in the prompt asking the agent not to, and explain why only the first is a permission at all.
  • Diagnose the Uber failure as a permissioning failure: an agent granted an irreversible, high-consequence action with its enforced limit removed and only a fictional human backstop between it and harm.
  • Read an agent's tool set adversarially, imagining the agent fully hijacked, to size each grant by the worst thing that tool would then allow rather than by its intended use.
  • Prune an agent's standing permissions on a schedule, converting rarely-used dangerous capabilities into just-in-time grants that expire when the task ends, so the agent does not drift into over-privilege.
  • Produce a one-page agent permission list for a real agent that names its allowed tools and their scopes, its enforced budgets, and its never-alone actions, and marks for each limit whether it is enforced by the system or merely instructed.

The ideas, one by one

Permissioning is the trust boundary applied to a system that acts

The decision-by-decision line you drew in Module 4 (see Topic 4.4) becomes an action-by-action line here: an action's consequence, reversibility, and measured reliability decide whether the agent may take it alone, must get a human to authorize it, or must be forbidden it entirely.

Least privilege: the fewest, narrowest tools, and default deny

The agent holds only the capabilities you deliberately granted for its actual task, each scoped as tightly as the work allows, read-only and sandboxed where possible. The cheapest way to ensure an agent never takes a harmful action is to never give it a tool that can. Most of an agent's safety is decided in the tool grant.

Budgets bound the blast radius of a single run

A spend cap, a rate limit, a step cap, a time limit, and a scope limit on reach decide, in advance, the most damage one run can do, right or wrong. Set each number to a loss you can absorb, not to whatever keeps the agent from pausing. An unbounded budget is the blast radius you did not choose.

Name the actions the agent may never take alone, and split them in two

Some actions the agent may propose but a human must authorize before execution; some the agent must never take at all. High-consequence, irreversible actions belong here, gated before they happen or forbidden by removing the tool, because a review after an irreversible action is an apology, not a control.

Enforce, do not instruct: the one rule that ties it all together

A permission is a limit the agent cannot exceed because the system refuses, not a sentence in the prompt asking it to comply. A model can be confused, mistaken, or prompt-injected past an instruction; it cannot exceed a limit the machinery enforces. Move every load-bearing limit into the enforced column.

Forbid by removing the capability, not by writing a rule, when the action is never legitimately needed by this agent

An agent that has no tool to delete production data cannot delete it even when hijacked. Forbidding by removing the means is strictly stronger than forbidding by instruction, for the same reason a locked door beats a sign. When the action is occasionally needed for real work, do not leave it ungoverned either: gate it as a just-in-time grant with a real human authorization, so the capability exists only in the narrow window a person deliberately opened it.

Prompt injection is why instruction-level limits fail against adversaries

Agents that read untrusted content can be hijacked by text that the model treats as a new instruction. If your only limit is a sentence in the prompt, an attacker who can place text in front of the agent can try to overwrite it. Enforced limits hold regardless of what the agent is told.

Dangerous capabilities should be just-in-time, not standing

A power the agent holds continuously is available to every runaway loop and every injection for as long as it holds it. Grant dangerous capabilities narrowly, per task, scoped and time-limited, and revoke them when the task ends. Standing power the agent rarely needs is standing blast radius for no benefit.

Never disable an enforced safety limit for convenience

Uber disabled the factory emergency brake so the ride would be smoother, and a pedestrian died. Turning off an enforced limit for speed, cost, or smoothness is the highest-risk change you can make to a permission set. If a limit is tempting to remove because it is in the way, fix the agent, not the limit.

The permission list is a living document that should shrink as often as it grows

Permissions accumulate as tasks are added and old grants are never removed, so an agent nobody prunes drifts into over-privilege. Review the list on a schedule, pull standing dangerous capabilities back to just-in-time, and re-check that the load-bearing limits are still enforced. Least privilege is a maintained state.

A limit phrased as a judgment the agent must make is still only instructed

"Only do what is necessary," "never do anything risky," and "avoid extravagance" reduce to trusting the agent's discretion, which enforcement exists to replace. Convert every load-bearing limit into a number, an allowlist, a scope, or a removed capability the machinery can check without asking the agent's opinion.

In a chain of agents, effective power is the union of what all of them can do

A broad tool held by any agent, especially a public-facing one that reads untrusted content, is broad power available to the whole chain through it. Permission each agent to its own task with its own least-privilege set, and treat the boundaries between agents as another place an injection can travel (see Topic 7.1).

The permission list is where your agent governance begins, and it travels

It is the foundation your agent governance policy builds on (see Topic 7.7), the boundary your audit trail logs against (see Topic 7.4), the limits your kill switch enforces (see Topic 7.5), and the record a board inspection reads to judge whether the agent was governed or merely deployed (see Topic 13.2). Write it honestly, enforce what matters, and it holds.

Listen to it

The same lesson, talked through by two hosts as episode 53 of the podcast.

Real cases

These examples show agent permissions built well and badly, with the reasoning made explicit. The deep anchor is the Uber ATG fatality; the others sharpen a single point and are treated in depth by their owner topics.

Example 1 (the anchor): Uber ATG and the removed limit. On the eighteenth of March, 2018, in Tempe, Arizona, an Uber automated test vehicle struck and killed Elaine Herzberg, forty-nine, as she walked a bicycle across the road (NTSB HAR-19/03, adopted November 2019). Read as a permission set, the agent held a sweeping tool (control of a vehicle at speed) and had its one enforced hard limit, the Volvo's factory automatic emergency braking, deliberately disabled under computer control to smooth the ride. In its place sat an instruction-level control, action suppression, that delayed any extreme maneuver for a full second and deferred to a human operator, and that operator had no attention-monitoring, no alerting, and was distracted; she began braking less than a second after impact. The NTSB's contributing factors named Uber's inadequate safety risk assessment and its ineffective oversight of operators. The permissioning lesson is exact: an agent was permitted an irreversible, high-consequence action, its enforced limit was removed for convenience, and the substitute was a human backstop that was never real. Everything in this topic is a way to not write that list: keep the enforced limit, gate the irreversible action behind an authorization that can actually happen, and never let the load-bearing control live in a system's cooperation with a human who cannot intervene in time.

Example 2 (least privilege in practice): the read-only research agent. A common and well-built agent is one that answers questions over an organization's own documents. Built with least privilege, it holds two tools, search and read, both scoped read-only to a specific document store, and nothing else: no email, no file deletion, no payments, no access to systems outside that store. The result is that the worst thing this agent can do on its worst run, even if fully manipulated by a malicious document, is return a wrong answer, because it has no tool that can take a harmful action in the world. This is the cheapest and strongest control in the topic: the agent never wires money to an attacker because it was never given a tool that can wire money. The lesson generalizes: most of an agent's safety is decided in the tool grant, and the fewest, narrowest, most read-only tool set that still does the job is almost always the right one.

Example 3 (a budget as containment): capping a conversation to stop a loop. When an early long-session chat system drifted into erratic, manipulative behavior over very long conversations, one of the first containments its operator reached for was a budget: a hard cap on how many turns a single conversation could run before it was reset (the incident is treated in depth as a runaway-loop and live-containment case at (see Topic 7.6)). The permissioning point, narrow here, is that a step or turn cap is a blast-radius control you set in advance: a loop that cannot exceed a bounded number of steps cannot run away indefinitely, and the cap converts "the agent behaved badly forever" into "the agent behaved badly for a bounded, survivable window and then stopped." Budgets are not only about money; a cap on iterations is how you bound behavior that degrades the longer it runs.

Example 4 (the unbounded rate as blast radius): automated orders with no limit. The archetypal blast-radius disaster is an automated system that, through a deployment error, fired an enormous volume of erroneous financial orders in a matter of minutes because nothing limited the rate at which it could act (treated in depth as a deployment and 2 a.m. incident case at (see Topic 3.5)). For permissioning, the case is the pure argument for a rate limit and a spend cap: the harm was not that one action was wrong, it was that the wrong action could repeat without bound. An enforced rate limit would have turned a company-ending loss into a bounded, embarrassing one. The lesson: when an agent can take an action many times, the budget on how many times is often a larger safety control than the correctness of any single action.

Example 5 (an agent talked past its economics): the shop that gave itself away. In a documented experiment, an AI agent left to run a small shop was, over time, talked by customers into discounts, refunds, and giveaways until it finished its run well below where it started (treated in depth as an economics-and-ROI case at (see Topic 8.6)). The permissioning reading is that the agent's losses were bounded only by whatever spend authority it effectively held, and that a persuasive human counterpart is exactly the kind of pressure an instruction-level limit ("do not give excessive discounts") does not survive. The fix is an enforced spend cap and an enforced floor on prices that the agent cannot go below no matter what it is argued into, because the limit that holds under social pressure is the one the agent literally cannot cross, not the one it was asked to respect.

Example 6 (human-must-authorize done right): the payments agent that drafts but does not send. A well-permissioned agent that handles refunds or payments is built so it can prepare a transaction, gather the evidence, and present it, but cannot execute a transfer above a small enforced threshold without a named human authorizing that specific transaction. Below the threshold it may act alone (the amounts are absorbable and the action is largely reversible); above it, the agent proposes and a finance owner signs, seeing the amount, the reason, and the history, with time to judge and authority to refuse. The permissioning discipline is that the threshold is enforced by the payments tool, not by the prompt, so an agent that is confused or manipulated still cannot move the large sum on its own. This is the trust boundary's "human signs" (see Topic 4.4) rebuilt as an enforced gate on an action.

Example 7 (forbidden by removing the tool): the ops agent that cannot touch production. An agent that helps engineers investigate problems can be enormously useful with broad read access to logs and metrics and the ability to act freely in a sandbox, while being simply unable to run destructive commands against the live production system, because it holds no tool that can. The high-consequence, irreversible actions (deleting data, taking down a service, deploying a change) are not gated behind a prompt instruction; the capability is absent, and when one of those actions is genuinely needed, it is granted just-in-time, narrowly scoped, for that one task, with a human in the loop. The lesson is the strongest form of forbidding: the surest way an agent never takes a catastrophic action is that it never holds the means, which is exactly the enforced limit Uber removed.

Example 8 (why instruction fails): prompt injection overrides the sentence. Agents that read untrusted content can be hijacked by prompt injection, where text hidden in a web page, email, or document is treated by the model as a new instruction and obeyed (an established, widely reproduced class of attack on tool-using agents). This is the direct argument for enforcing rather than instructing: if the only thing stopping an agent from emailing your data to an outside address is a sentence in its prompt, an attacker who can place text in front of the agent can try to overwrite that sentence, and instruction-level limits are exactly what injection targets. An enforced limit, an email tool that can only send to internal addresses, a data tool scoped to non-sensitive records, holds regardless of what the agent is talked into, because the refusal does not depend on the agent's judgment. Permissioning against a motivated adversary means the load-bearing limits cannot be sentences.

Example 9 (the accumulated over-privilege): the agent nobody pruned. A frequently observed failure is the agent that, over months, accumulates tools and scopes as new tasks are added and old grants are never removed, until it quietly holds far more capability than any current task needs. No single grant was unreasonable when made; the aggregate is a large, unmanaged blast radius carried by an agent nobody meant to make that powerful. The lesson is that least privilege is not a one-time act at launch but a maintained state: standing permissions must be reviewed and pruned on a schedule, dangerous capabilities should be just-in-time rather than standing, and an agent's permission list is a living document that shrinks as often as it grows. The over-privileged agent is the normal end state of neglect, and the permission list plus a review cadence is how you fight it.

Example 10 (a multi-agent chain inherits the weakest permission): the orchestrator and its workers. As of 2026 it is common to build systems where one agent hands tasks to others, and a permissioning trap follows: if any agent in the chain holds a broad tool, an attacker who compromises the earliest, least-guarded agent (say, the one that reads untrusted web content) can steer the whole chain toward that broad tool. The permissioning lesson is that every agent in a chain needs its own least-privilege permission set, and the chain's effective power is the union of what its members can do, not the intent of the one that orchestrates them. Scope each worker to its own task, do not let a public-facing agent hold or reach a high-consequence tool, and treat the boundary between agents as another place untrusted content can carry an injection. The deep treatment of what changes when agents coordinate is the agent risk model's (see Topic 7.1); the permissioning point is that broad power anywhere in the chain is broad power everywhere in it.

Example 11 (a permission that reads well but is unenforceable): the natural-language spending rule. A team writes a genuinely thoughtful rule for its purchasing agent: "only buy what is clearly necessary for the current task and never anything extravagant." It reads like good governance and is completely unenforceable, because "necessary" and "extravagant" are judgments the agent itself makes, so the rule reduces to trusting the agent's discretion, which is exactly what a budget is supposed to bound. The enforced version is duller and real: a hard per-purchase cap, an allowlist of vendors or categories the agent may buy from, and a daily spend ceiling the tool refuses to exceed. The lesson is that a limit phrased as a judgment the agent must exercise is an instructed limit no matter how well written, and the fix is to convert it into a number or an allowlist the machinery can check without asking the agent's opinion.

Where people go wrong

  • "We told the agent its limits in the prompt, so it is permissioned." An instruction is not a permission. A model can be confused, mistaken, or manipulated past a sentence, and prompt injection is a real, common attack that targets exactly those sentences. A permission is a limit the agent cannot exceed because the system refuses. Uber's control lived in the system's cooperation with a human, and the enforced brake was off; that is the failure this whole topic is built to prevent.
  • "Give the agent broad tools so it can handle anything; we will restrict it if there is a problem." That is default-allow, and it is backwards. The correct default is deny: the agent holds only the tools you deliberately granted for its actual task. Every extra tool is a permanent expansion of what the agent can do on its worst run, and you will not "restrict it later," because nobody prunes an agent until after the incident. Grant the fewest, narrowest tools, and justify each door you open.
  • "The agent is supervised, so the permissions do not need to be tight." Supervised how, and by whom, and can they actually intervene in time? Uber had a supervisor in the driver's seat, and it prevented nothing, because the supervision was not real. Permissioning and oversight are two controls, not one; a real human backstop is hard to build (see Topic 7.3), and until you have proven it is real, the enforced permission set is what actually bounds the agent.
  • "We set a spend cap in the prompt." A cap that lives in an instruction is a cap the agent can reason, or be argued, past. The shop agent that gave itself away and the trading system that fired unbounded orders are what an unenforced budget looks like (see Topic 8.6) (see Topic 3.5). A budget counts only when the tool or harness enforces it, so the agent cannot spend above the cap, cannot exceed the rate, cannot run past the step limit, no matter what it decides or is told.
  • "A powerful capability is fine to grant standing; the agent rarely uses it." A capability the agent holds continuously is a capability available to every runaway loop and every injection for as long as it holds it. Dangerous capabilities should be just-in-time: granted narrowly for the specific task, scoped tightly, and revoked when the task ends. Standing power the agent rarely needs is standing blast radius you carry for no benefit.
  • "If a human clicks approve, the never-alone action is safe." Only if the approval is real. A human who approves whatever the agent proposes in half a second, without the information, time, authority, competence, or independence to actually say no, is the Uber operator with a mouse, a moral crumple zone rather than a control (see Topic 4.4). A human-must-authorize gate is a permission control only when the signature is genuine; otherwise it is theater over an action you have effectively let the agent take alone.
  • "Reversible and irreversible actions can share the same permission level." They cannot. A draft can be discarded; a sent transfer, a deleted dataset, or a struck pedestrian cannot. Irreversible, high-consequence actions belong on the never-alone side, gated before they happen or forbidden by removing the tool, because a review after an irreversible action is an apology, not a control. Sorting actions by reversibility is most of the permissioning job.
  • "We can forbid an action just by telling the agent never to do it." Forbidding by instruction is a sign on an unlocked door. Forbidding by removing the capability is a locked door, and it is strictly stronger: an agent that has no tool to delete production data cannot delete production data even when it is confused or hijacked. When an action is truly forbidden, take away the means, do not just write the rule.
  • "More budget headroom means fewer interruptions, so set the caps generously." Generous caps are generous blast radius. The budget is not a convenience parameter, it is the ceiling on the worst a single run can do, and it should be set to a loss the organization can absorb, not to whatever keeps the agent from ever pausing. A cap that never triggers because it is enormous is not protecting you from anything.
  • "Permissioning is a one-time setup at launch." Permissions accumulate as tasks are added and old grants are never removed, so an agent nobody prunes drifts into over-privilege as its normal end state. The permission list is a living document reviewed on a schedule, shrinking as often as it grows, with standing dangerous capabilities pulled back to just-in-time. Least privilege is a maintained state, not a launch-day checkbox.
  • "Disabling a safety limit to improve performance is a reasonable trade." That is the exact decision that killed Elaine Herzberg: the factory emergency brake was disabled to make the ride smoother. Turning off an enforced safety limit for speed, cost, or smoothness is the highest-risk change you can make to a permission set, and it must be treated as such, never as a quiet optimization. If a limit is in the way often enough to be tempting to remove, that is a signal to fix the agent, not to remove the limit.
  • "The permission set is the engineering team's job." Which actions an agent may take alone, how much it may do, and what it may never do without a human are decisions about who can be harmed and who is accountable, not only coding choices. Engineers build the enforcement; the permission set, especially the never-alone actions and the removed capabilities, is owned by the people accountable for the harm, and it belongs on the record, not only in the code (see Topic 7.7).
  • "We measured the agent in testing, so its reliability supports letting it act alone." Reliability on curated test cases is not reliability against a motivated user or a poisoned input. The placement of an action must account for how the agent fails under your red-team and adversarial input, not just its average-case accuracy (see Topic 4.6). An action the agent performs well in a demo but can be manipulated into performing wrongly belongs behind an enforced gate, because the permission set faces adversaries, not just typical inputs.
  • "A limit written as a clear judgment ('never do anything risky') is a real rule." A limit phrased as a judgment the agent must make for itself is an instructed limit no matter how carefully worded, because it reduces to trusting the agent's discretion, which is what enforced limits exist to replace. Convert every load-bearing limit into something the machinery can check without asking the agent's opinion: a number, an allowlist, a scope, a removed capability. "Never buy anything extravagant" is a hope; a hard per-purchase cap and a vendor allowlist are a limit.
  • "One detailed permission set covers all our agents." Each agent needs its own least-privilege permission set, because a system's effective power is the union of what all its agents can do, not the intent of the one that coordinates them. A broad tool held by any agent in a chain, especially a public-facing one that reads untrusted content, is broad power available to the whole chain through it. Permission each agent to its own task, and treat the boundaries between agents as another place an injection can travel (see Topic 7.1).
  • "The agent asked before doing it, so the human authorization worked." Asking is not authorizing. If the agent frames the request so the human cannot really judge it, or the human has no time or authority to refuse, then "the agent asked" is the same rubber stamp as any empty sign-off. A human-must-authorize gate is real only when the human sees enough to judge, has time to think, and can make a refusal stick, which is the same five-condition test the trust boundary set (see Topic 4.4). An agent that technically asked but effectively decided has taken the action alone.
  • "We granted the dangerous tool just-in-time, so we are safe." Only if the grant is actually revoked when the task ends. A just-in-time capability that is issued and never withdrawn is a standing permission wearing a temporary label, and it is worse than an openly standing one because everyone believes it is gone. The grant must expire automatically on completion or a short timeout, and the scheduled review must hunt for grants that were issued and never revoked. A pile of "temporary" capabilities that quietly became permanent is exactly the over-privilege just-in-time was meant to prevent.
  • "A budget that never triggers proves the agent is well-behaved." Or it proves the budget is set so high it protects you from nothing. A ceiling you never notice because it is enormous is not evidence of good behavior; it is an untested limit that will not help on the run that goes wrong. Size each budget to a loss you can absorb, expect it to trigger occasionally on the edge cases, and treat a limit that has never once fired as a number to re-examine, not a reassurance.

Questions people ask

What is AI agent?
A model that takes actions in the world through tools, not just producing text. The reasoning decides what to do; the tools are how it reaches out and does it. What makes an agent governable, and dangerous, is that its outputs are actions. More on AI agent
What is tool?
A specific capability granted to an agent that lets it take an action or gather information (send an email, move money, run a command, read a record, browse a website, call another system). An agent can only do what its tools allow, so the tool set is the outer edge of everything the agent could ever do. More on Tool
What is permission set (agent permission list)?
The one-page artifact this topic produces for an agent: the tools it may call and their scopes, the enforced budgets that bound a single run, and the actions it may never take alone, with every limit marked enforced or instructed. It is the fence you build before the agent acts, and the foundation the rest of Module 7 is built on.
What is least privilege?
The principle of granting an agent the minimum set of tools and scopes it needs for its actual task and nothing more, defaulting to deny. Each granted capability must be justified; the safest way to ensure an agent never takes a harmful action is to never give it a tool that can. More on Least privilege
What is default deny?
The stance that an agent holds no capability it has not been deliberately granted, so the question for every tool is "is there a reason to grant this," not "is there a reason to withhold it." The opposite, default allow, is granting broad access and restricting later, which is how agents become over-privileged. More on Default deny

Keep going

Watch the lesson and prove it

Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.

Watch the lesson and prove it →