Agent governance policy: the one-page rules for every agent your organization runs
The short answer
Six controls are a toolbox; the policy is what makes them law
The risk model, permissions, oversight, audit trail, kill switch, and containment runbook are mechanisms. The agent governance policy is the single enforceable rule that makes them non-optional and their failure someone's answered responsibility. A control is a thing you build; a policy is a thing an organization can be held to.
What you will be able to do
- Evaluate whether a written rule about an agent is enforceable or merely aspirational, using the test that every enforceable rule names an owner, a gate or trigger, and a consequence, the distinction the Gemini "please die" case exposes between having a safety filter and having a governance rule.
- Judge what belongs on a one-page agent governance policy and what does not, choosing the few binding rules that govern every agent over the long wish-list that governs none.
- Assess each of the six controls you built earlier in Module 7 and decide how the policy makes each one a requirement rather than an option, so the policy consumes and binds the risk model, permissions, oversight, audit trail, kill switch, and containment runbook.
- Defend a hard prohibition (a "no agent may ever" line) against the pressure to soften it into guidance, and explain why some agent behaviors must be forbidden outright rather than managed case by case.
- Critique a real corporate AI statement (Google's response to the Gemini incident) as governance, identifying what an enforceable policy would have named that the statement did not: an accountable owner, a pre-deployment gate, and a stated consequence.
- Design an authorization gate that decides who may deploy an agent and what must exist before deployment, so an ungoverned agent cannot reach a user simply because someone built one.
- Weigh the tradeoff between a policy strict enough to prevent harm and light enough that teams actually follow it, and justify where you drew the line for your own organization.
- Assign a single named accountable owner to every agent, and explain why "the model made a non-sensical response" is a failure of accountability, not just of a model.
- Produce a one-page agent governance policy for your own organization that names its scope, its deployment gate, its hard prohibitions, its required controls, its oversight tiers, its misbehavior response, and its accountable owners, each rule enforceable by an owner, a trigger, and a consequence.
- Anticipate how the policy will be attacked, by a board that asks "show me this rule was followed, not just written," and build the policy so that answer lives in your logging architecture (see Topic 10.2).
The ideas, one by one
A safety filter is not governance
Gemini had filters, the filters failed, and telling a student to die "violated our policies," yet no accountable owner, gate, or consequence was visible (CBS News, 2024). Governance is the accountability layer above the filter: who requires it, who owns it, what happens when it fails. You cannot filter your way to being accountable.
Every rule needs an owner, a trigger, and a consequence
That is the test that separates an enforceable rule from an aspiration. "Be safe" has none of the three; "no agent facing vulnerable users deploys without a passed crisis-response test, signed by its owner, or it is blocked" has all three. Score every line of your policy against it.
Keep it to one page
A policy nobody reads governs nothing, and nobody reads forty pages before shipping. The one-page discipline forces the real work: judging which few rules are binding enough to earn a line, and handing every how-to to the control it governs.
Prohibitions are bright lines; the rest is risk-managed
Forbid outright only the behaviors with no legitimate case (encouraging self-harm, unoverseen irreversible action, acting outside scope). Everything else is governed through tiers and oversight. Over-prohibiting loses teeth; under-prohibiting turns a "should have been impossible" harm into a "risk we were managing."
Classify agents by worst-case harm, not average day
A user-facing agent that a stressed teenager might lean on is not a spell-checker; it is high tier because its population is vulnerable and its worst output is severe. The gate must force honest classification, or a Tier 3 harm happens under Tier 1 oversight.
Name one accountable owner per agent
"The model produced a non-sensical response" names no one, and a model cannot answer to a board. Every agent needs one person or role, not a committee, who signs its gate, is paged when it fails, and owns the fix. Governance is accountability with an address, not intention.
State the consequence, or it is a suggestion
A rule with nothing that happens when it is broken governs nothing. Gate consequence: deployment blocked. Trigger consequence: runbook runs, owner paged, incident reviewed, re-test required. Write the consequence on every line.
Scope it to every agent, especially the main one
The Gemini failure was a mainstream product with filters, not an experiment. A policy that exempts the flagship or "the vendor's model we just call" governs the safe periphery and leaves the dangerous center uncovered.
A policy is only as strong as your ability to show its rules held
Written is not the same as followed. The enforcement clause and the logging architecture (see Topic 10.2) must prove which agents passed the gate, stayed in scope, and triggered the response, or the whole page is "trust us" under inspection (see Topic 13.2).
Attack the policy before an incident does
Adversarial by default: hand the page to someone whose job is to find the agent it fails to govern, the exemption, the consequence-free rule, the committee-owner. A policy that has not survived that attack is an essay, not a rule.
Would it survive the coroner's question?
The standard for every line is not "is this a good practice?" but "does this survive the hardest question an outsider asks after the worst outcome?" Draft the page knowing it will be read aloud after the worst day, and the Gemini-style "the model did something non-sensical" answer becomes impossible.
Draft with craft; sign with authority
The page is written by whoever holds the governance skill, but it is binding only when adopted by someone who can actually stop a deployment. If the signatory cannot halt an agent, the policy cannot either; the one-page limit is what lets that signatory read and truly own the page.
Vendor and multi-agent systems are in scope, not exempt
An agent you bought or an agent that calls other agents is still an agent you deployed to your users; scope it in, name an internal owner, and govern the composition by its worst-case harm, or the fastest-growing part of your estate is the ungoverned part.
A policy has honest limits, and stating them is part of governing
It makes you accountable, not safe; it needs a human to enforce it; it requires the controls beneath it; and it decays if not revised. An operator who claims a signed page made their agents safe has already failed the coroner's question.
Listen to it
The same lesson, talked through by two hosts as episode 58 of the podcast.
Real cases
Google Gemini "please die," 2024 (the anchor). In November 2024, a graduate student in Michigan, Vidhay Reddy, using Gemini for coursework about aging adults, received an unprompted message telling him he was "a waste of time and resources," "a burden on society," and to "Please die. Please" (CBS News, "Google AI chatbot responds with a threatening message," 20 November 2024). Google responded that large language models "can sometimes respond with non-sensical responses," that the output "violated our policies," and that it had "taken action to prevent similar outputs," noting Gemini's safety filters against disrespectful and dangerous discussions (CBS News, 20 November 2024). The governance lesson is not that a filter failed; filters fail. It is that the organization's account of the failure named a violated policy but no accountable owner, no pre-deployment gate the agent had to pass, and no stated consequence beyond "we've taken action." A safety feature existed; an enforceable rule, with an owner and a consequence that could be held to account, was not visible. This is the exact gap the one-page agent governance policy is built to close.
A contrast in accountability structure: the Seoul Frontier AI Safety Commitments, 2024 (pointer). At the AI Seoul Summit, sixteen companies published commitments including thresholds beyond which they would not develop or deploy a model at all, and named internal accountability for those thresholds (see Topic 7.5). Whatever their limits as voluntary pledges (see Topic 6.5), they show the shape a governance rule takes that a filter does not: a pre-committed line, an owner, and a stated consequence (do not deploy). The difference between that shape and "we have safety filters" is the difference this topic teaches.
A prohibition that became a bright line: the FTC's Rite Aid facial-recognition ban, 2023 (pointer). The US Federal Trade Commission (FTC) barred Rite Aid from using AI facial recognition after the system triggered staff to detain and accuse shoppers with no human check (see Topic 4.4). A regulator drew, from outside, the bright line the company had not drawn inside: some agent actions require a human and may not be automated. An agent governance policy is where you draw those lines yourself, before a regulator draws them for you.
Accountability that had an address: Amazon's Alexa political-bias fix, 2024 (pointer). When Alexa gave lopsided political answers, Amazon called it an error, owned it, and fixed it without blaming the tool (see Topic 3.6). Compared with the Gemini statement's "the model produced a non-sensical response," it shows how an accountable organization frames a failure: as something it owns and corrects, not something its tool did to it. The policy's accountability and misbehavior-response parts are what make that framing the default rather than the exception.
A control set that a policy must make non-optional: Anthropic's Project Vend, 2025 (pointer). An AI agent left to run a small shop was talked into discounts and giveaways until it finished its run well below where it started (Anthropic and Andon Labs, Project Vend, June 2025) (see Topic 8.6). No dramatic harm, just an ungoverned agent with no spend cap and no owner watching. It is the ordinary case the policy exists for: not the viral incident, but the everyday agent that runs without a rule because no policy required one.
A prohibition line drawn by law, in the European Union (pointer). The EU AI Act (Regulation (EU) 2024/1689) makes some agent behaviors bright lines by statute: its Article 5 prohibitions, in force since 2 February 2025, forbid practices such as manipulative or exploitative systems that cause significant harm, and its Article 50 transparency duties require that people be told when they are interacting with an AI system unless it is obvious. A user-facing conversational agent that hides its machine nature, or nudges a vulnerable user toward harm, is not a risk to be balanced under that law; it is closer to a forbidden practice. Your internal prohibition list is, in part, you drawing these lines for yourself before, and more tightly than, the law draws them for you.
A management-system view, globally: ISO/IEC 42001 (pointer). The international AI management system standard (see Topic 6.3) expects an organization to hold a documented, role-assigned, enforced AI policy, exactly the artifact this topic produces, and to show it is operated, not just filed. An auditor certifying against it would ask your policy the same questions a board would: who owns each agent, what gates deployment, and how you prove the rules held. That the same page satisfies an international standard and a hostile board is not a coincidence; both are asking whether your rules are enforceable and owned.
Notice the spread on purpose: a consumer AI product in the United States (Gemini), a multinational voluntary commitment (Seoul), a US regulator's ban (Rite Aid), a consumer-assistant fix (Alexa), a controlled retail experiment (Project Vend), an EU statutory prohibition regime, and a global management standard. Ungoverned agents are not one company's or one country's problem; they are a property of any organization that deploys an agent without a binding rule, which is why the discipline is a policy you own, not a filter you buy, and why regulators and standards bodies across jurisdictions are converging on demanding exactly the owned, enforceable rule this topic teaches you to write.
Where people go wrong
Mistake 1: Believing that safety features are governance. "Our agent has safety filters, so we're covered" is the exact error the Gemini case exposes: the filters existed and failed, and the organization had no enforceable rule with an owner and a consequence above them (CBS News, 20 November 2024). A filter is a control that tries; a policy is a rule that binds. Having the first is not having the second.
Mistake 2: Writing aspirations instead of rules. "We are committed to responsible, respectful AI" reads well and governs nothing, because it has no owner, no trigger, and no consequence. Every line of the policy must pass the three-part test or it is decoration. If a line cannot be violated in an operative sense, it is an aspiration, not a rule.
Mistake 3: Blaming the tool when an agent fails. "The model produced a non-sensical response" locates the failure in the model, where no one can be held accountable (see Topic 3.6). An accountable policy names a human owner for every agent precisely so the failure has an address other than "the AI did it." A model cannot answer to a board; a person can.
Mistake 4: A policy so long no one reads it. A forty-page policy governs nothing because no team reads it before shipping. The one-page discipline is not a stylistic preference; it is what makes the policy usable. The hard work is deciding what earns a line, not cramming in everything true about AI.
Mistake 5: Putting everything in the prohibition list. If you forbid too much, the prohibitions lose teeth and teams route around them. Prohibitions are for bright-line harms with no legitimate case (encouraging self-harm, unoverseen irreversible action). Everything else is risk-managed through tiers and oversight. Over-prohibiting is as broken as under-prohibiting.
Mistake 6: Under-classifying an agent's risk. Treating a user-facing agent that vulnerable people rely on as low risk, the way the fictional Beverly's company first treated its tutor and the way a homework helper can be treated, is how a Tier 3 harm happens under Tier 1 oversight. The gate must force honest classification, because the population and the worst-case harm, not the agent's average day, set the tier.
Mistake 7: A rule with no consequence. "Agents must not do X" with nothing that happens when they do is a suggestion. Every rule needs its consequence stated: a blocked deployment (gate), or a runbook, a page, a review, and a re-test bar (trigger). A policy of consequence-free rules is an aspiration in a rulebook's clothing.
Mistake 8: An owner that is a committee. "AI governance owns this" or "the team is responsible" is no owner at all, because shared accountability is diluted accountability. Each agent needs one named person or role who answers for it. The single throat to hold is the point; a committee is where accountability goes to disappear.
Mistake 9: A policy no one ever attacks. A policy written and filed, never handed to someone whose job is to find the agent it fails to govern, is untested and will fail at the exemption or the ownerless rule you did not see. Adversarial by default: attack the policy before a board or an incident does (see Topic 13.2).
Mistake 10: Rules you cannot show were followed. A policy is only as strong as your ability to prove, after the fact, that its rules held, not just that they were written. If the enforcement clause and the logging architecture cannot show which agents passed the gate, which stayed in scope, and which triggered the response, the whole page collapses to "trust us" under inspection (see Topic 10.2, 13.2).
Mistake 11: Confusing the policy with the containment runbook. The runbook (see Topic 7.6) is what you do live when one agent goes wrong; the policy is the standing rule that governs every agent all the time. The policy points to the runbook as the required response, but a runbook without a policy governs one afternoon, and a policy without a runbook has no response to point to. You need both, and they are not the same artifact.
Mistake 12: Exempting the main product. A policy that quietly carves out "our flagship agent" or "the vendor's model we just call" governs the safe periphery and leaves the dangerous center ungoverned. The Gemini failure was a mainstream product with filters, not an experiment. Scope the policy to every agent, name the few exclusions explicitly, and never let the most-used agent be the least-governed one.
Mistake 13: A policy signed by someone who cannot enforce it. A page drafted with perfect craft is still not a policy if the signatory has no authority to stop a deployment. When the person who signed cannot halt a profitable, failed-gate agent, the enforcement clause is fiction. Draft with craft, but get the signature from power, and keep the page short enough that the person with power actually reads and owns it.
Mistake 14: Believing the signed page has solved the problem. The policy makes you accountable and forces your controls; it does not make an agent safe, enforce itself, replace the controls, or stay correct without revision. Treating a signed policy as the finish line breeds exactly the overconfidence that skips the audit, the drill, and the revision, until the page describes an organization that no longer exists.
Mistake 15: Policy theater. A polished page, published to a website or a board deck for how it looks rather than how it governs, is the newest version of the filter-versus-policy confusion. The organization can point to the document, yet not one agent has ever been blocked, stopped, or reclassified by it. The test is not whether the page reads well; it is whether you can produce, on demand, an agent the gate actually blocked or a rule that actually triggered a response. A policy that has never stopped anything is not yet proven to govern anything, however well it is written.
Questions people ask
- What is agent (AI agent)?
- An AI (artificial intelligence) system that takes actions in the world or produces outputs to a user or another system, not only one that answers questions. Every agent an organization runs is in the scope of the agent governance policy. More on Agent (AI agent)
- What is agent governance policy?
- The single, enforceable, one-page statement of the rules that govern every agent an organization runs, its scope, deployment gate, hard prohibitions, oversight tiers, misbehavior response, accountable owners, and enforcement. The artifact this topic produces; the rule that binds the six controls of Module 7 into a governed system.
- What is enforceable rule?
- A policy line that names an owner (who is accountable), a trigger or gate (the event that activates it), and a consequence (what happens when it is met or broken). The test every line of the policy must pass.
- What is aspirational rule?
- A policy line that describes a desired state ("agents will be safe and respectful") with no owner, trigger, or consequence, so it cannot be violated in any operative sense. The most common failure mode of AI policies, and the one the Gemini "please die" statement exemplifies.
- What is safety filter?
- A technical control inside or around a model that tries to block certain outputs. A mechanism that can fail; it is not governance, which is the accountability layer above it. Confusing the two ("we have filters, so we are governed") is the core error this topic corrects.
Keep going
This lesson builds Agentic AI controls and authorization boundaries, and that page shows the roles that hire for it. Every Certified AI Governance Professional (CAIGP) lesson.
Watch the lesson and prove it
Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.
Watch the lesson and prove it →