Skip to main content

Build, buy, or kill: the quarterly portfolio review with real numbers

The short answer

A portfolio review judges every system together, on a schedule

You cannot see which system is quietly rotting or quietly carrying the business by staring at one; you see it by lining them all up. A fixed quarterly cadence inserts a cold look in the calm middle of a system's life, between the optimism of launch and the fury of failure, which is the only time an honest verdict is possible. Skip the schedule and the only reviews you ever get are the launch and the crisis.

What you will be able to do

  • Run a quarterly portfolio review of your organization's AI systems: take the AI systems inventory you built in Module 0, put a real number on each system's cost and return, and produce one decision (build, buy, or kill) for every system in the same sitting.
  • Distinguish the three verdicts precisely: build (keep investing to develop or scale a system in-house), buy (replace or acquire the capability from a vendor), and kill (stop and decommission), and name the fourth quiet option (hold steady) for what it usually is, a weak build that is avoiding a real decision.
  • Calculate a system's honest total cost of ownership by adding the build cost, the run cost, the supervision tax you measured in Topic 8.2, and the risk cost, and set it against the honest return you established in Topic 8.1. (see Topic 8.1) (see Topic 8.2)
  • Evaluate a kill decision on its merits by separating the sunk cost (money already spent, which is gone whatever you decide) from the forward cost (what keeping it alive will cost from today), and recognize escalation of commitment when you or a colleague argue from the sunk cost.
  • Set a kill gate: a pre-committed threshold, agreed before anyone is attached to the system, that converts a future kill from an emotional fight into a decision that was already made.
  • Judge the cost of a late kill: why killing a system at a gate before deployment is cheap and killing it after it has shipped and harmed people, as Ofqual was forced to, is catastrophic, and why a reversible system can be scaled boldly while an irreversible one must be shipped slowly.
  • Defend each verdict in your portfolio review to a hostile board with the numbers behind it, so the review becomes the backbone of the investment memo in Topic 8.6 and survives the board inspection in the capstone. (see Topic 8.6) (see Topic 13.2)

The ideas, one by one

Every system leaves with exactly one verdict: build, buy, or kill

Build is a decision to spend more and must earn it; buy trades control for a lower total cost of ownership but never removes your oversight duty; kill stops and decommissions. "Hold steady" is not a fourth verdict; it is a build too timid to name its budget or a kill too attached to say the word. A deferred decision on a failing system is how you kill it too late.

Run it on two honest numbers, not impressions

Each system gets an honest total cost of ownership (build, run, supervision tax, risk cost) and an honest return (real value net of the work it creates). The supervision tax and the risk cost are the two most often omitted and the two most likely to flip a verdict. A review run on anecdote is theater; a review run on numbers is a judgment you can defend to a board. (see Topic 8.1) (see Topic 8.2)

Sunk cost is irrelevant, and saying so out loud is the skill

Money and time already spent are gone whatever you decide, identical on both sides, so they cannot belong in a live decision. When the argument for keeping a system leads with the past, it is escalation of commitment, and the cure is one flat question: if we were starting today, knowing what we know, would we begin this? MD Anderson's 62 million US dollar Watson project is what happens when that question is never asked.

Set the kill gate while you are cold

A pre-committed, specific, measurable threshold, agreed before anyone is attached, converts a future kill from a fight into the execution of a decision the team already made together. Ofqual's disaster read through this lens is the absence of a gate: the damning numbers were visible before results day, and with no line pre-agreed, the kill had to be improvised under a crowd four days after the harm.

The cost of a kill rises every gate you let a doomed system pass

Killing at the idea stage costs nothing; killing a deployed one-way door that people now depend on is catastrophe management, the Ofqual stage. So kill early or pay compounding interest to kill late. A gate is not a hoop, it is a cheap exit, and removing gates to move faster only guarantees every kill happens at the most expensive possible moment.

Reversibility decides how boldly you move

Scale two-way-door systems boldly, because being wrong costs a quick reversal; ship one-way-door systems slowly and kill them readily, because being wrong costs irreversible harm and no reversal can rescue you. Ofqual's algorithm was a one-way door dressed as routine: once grades issued, university places moved faster than any U-turn could catch.

A kill is a portfolio success, not a personal failure

Done at the right gate it frees capital, frees your scarcest reviewers, and removes uncompensated risk. Treating kill as failure is what makes kills late and therefore ruinous. Celebrate the reasoned kill, separate ending a system from judging its builders, and handle the human consequences with care rather than avoiding the kill to dodge them. (see Topic 9.1)

If you do not kill it, someone else will, on worse terms

A system that should die will be killed by the market, a regulator, the public, or a two a.m. incident if you do not kill it yourself. The review is how you keep the kill decision yours, made at a cheap gate on your calendar instead of at an expensive crisis on someone else's. That is why it is a governance instrument, not an accounting chore.

The biggest cost of a loser is the winner it starves

The opportunity cost rarely appears on a system's own row, and it is often the largest cost of all: the value of what the same money and, above all, the same scarce supervision could have done instead. "It is not costing us much" is usually wrong: even a cheap-to-run system consuming a reviewer is starving your best systems of the attention they need. A complete kill names where the freed resources go, because reallocation from losers to winners is the whole point of a portfolio.

A kill is a project, not a switch

The verdict is the decision; the decommission is the work, and it must handle the dependents, the data, the people whose roles change, and the record of why the system was killed. A kill agreed in the room but never assigned an owner and a plan is a kill that quietly keeps running, and a decommission done carelessly turns a good decision into a fresh incident. (see Topic 9.1)

The review is built to be defended

Its verdicts feed the investment memo a CFO stress-tests next, and the board inspects them line by line in the capstone. Every verdict must trace to a number a hostile reader can check, and every kill must be defensible on forward numbers alone. Write the reasons down now, because the board's hardest question, "why did you keep running that," must already have an answer that is a dated, gated decision. (see Topic 8.6) (see Topic 13.2)

Listen to it

The same lesson, talked through by two hosts as episode 62 of the podcast.

Real cases

These are real, documented cases. Each shows what a portfolio review is for by showing what a build, buy, or kill decision looks like when it is made well or made too late.

Example 1: Ofqual's A-level grading algorithm (England, 2020). This is the anchor. With exams cancelled by the pandemic, Ofqual built the Direct Centre Performance model to standardize teacher-submitted grades against each school's three-year history (BBC News, "A-levels and GCSEs: How did the exam algorithm work?", 2020). The method leaned on the historical pattern for classes of fifteen or more, left the smallest classes' teacher grades entirely untouched, and blended for the cohorts in between, which systematically advantaged smaller private-school cohorts over larger state-school ones. On results day, 13 August 2020, nearly 36 percent of A-level grades were one grade below the teacher assessment and about 3 percent were two grades lower (BBC News, 2020). After public protest, the government reversed course on 17 August and reverted to the teacher-assessed grades; the chief regulator resigned on 25 August. The portfolio lesson is exact and has two halves. First, the kill (revert to human grades) was the right verdict and was available, far more cheaply, before results day, because the damning numbers were visible in Ofqual's own testing. Second, because there was no pre-committed gate and no willingness to kill early, the decision was made four days after national harm, at the maximum cost, with the choice taken out of the regulator's hands by public fury. A quarterly review with real numbers and a fairness kill gate is the instrument designed to force that decision at the cheap gate instead of the expensive crisis.

Example 2: MD Anderson and IBM Watson for Oncology (United States, terminated 2016, audited 2017; referenced). The deep treatment of this case belongs to Topic 3.1, where it anchors the gap between a vendor's demo and a deployed reality (see Topic 3.1); here it carries exactly one lesson, the sunk cost trap. MD Anderson Cancer Center partnered with IBM to build a Watson-based system to guide cancer treatment. A University of Texas System audit found the project cost more than 62 million US dollars, was terminated while still in development, and never guided the care of a single patient (STAT News, 2017; University of Texas System audit, 2017). It is the standing example of the sunk cost trap: at each review the mounting sum already spent argued for spending more, which is the one argument that should carry no weight in a live kill decision. The forward question, "if we were starting today, would we begin this," would have said stop long before the total reached 62 million. (Marker: the audit figures are as reported by the University of Texas System and contemporary reporting; the teaching point about sunk cost stands regardless of the precise final number.)

Example 3: The build-or-buy decision made with real budget (referenced). The Michigan MiDAS unemployment system is the owned anchor of Topic 3.2, where a state built an automated fraud-adjudication system that, run without human oversight, wrongly accused tens of thousands of people. (see Topic 3.2) It is referenced here only to mark the through-line: the build-or-buy choice that topic teaches at a system's birth is the same choice this review reopens every quarter, now with the added verdict of kill that MiDAS itself so plainly earned and was denied for years.

Example 4: The honest ROI that a review must use (referenced). Topic 8.1 anchors the honest-return discipline to the Indian company Dukaan, whose chief executive publicly cut about 90 percent of support staff for a chatbot and claimed large cost savings. (see Topic 8.1) It is referenced here because the honest return you established there is one of the two numbers this review runs on. A portfolio review is only as truthful as the return figures fed into it; a headline saving that ignores the work the system created will keep a losing system alive.

Example 5: Scoping against reality before you build (referenced). Topic 3.1 anchors the gap between an AI system's demo and its real capability to the collapse of the telehealth firm Babylon Health, whose symptom checker never matched its hype. (see Topic 3.1) The link to this topic is direct: a system whose demonstrated value never materializes is a kill candidate at the first review, and catching that gap early, at a cheap gate, is precisely what a scheduled review with honest numbers is designed to do.

The pattern across these cases. Read together, the examples show one recurring shape, and naming it makes the review's purpose concrete:

  • The correct kill decision was almost always available earlier and cheaper than the moment it was actually made.
  • The loudest argument against killing was the money already spent, which is the one argument a sound decision must ignore.
  • Where the decision was delayed, it was not avoided; it was simply made later, by a regulator, a market, or a crowd, at far higher cost.
  • The systems that did the most harm were one-way doors, where by the time the failure was undeniable, the harm could not be reversed.
  • The biggest cost of the systems kept too long was rarely their direct run cost; it was the attention and supervision they drained from the systems that deserved it.
  • In every case, a scheduled review with real numbers and a pre-committed gate would have converted a public catastrophe into a routine line item.

That shape is why this topic exists: the review is the difference between "we ended it ourselves, early, on the numbers" and "it was ended for us, late, in the news."

Where people go wrong

  • "We already put so much into it, we cannot kill it now." This is the sunk cost fallacy, and it is the single most expensive sentence in AI governance. Money and time already spent are gone whatever you decide; they are identical on both sides of the choice and therefore irrelevant to it. The only costs that count are forward costs. The fix: whenever the argument for keeping a system leads with the past, ask the flat question, "if we were starting today, knowing what we know, would we begin this?" If the answer is no, the verdict is kill, and 62 million already spent, as at MD Anderson, changes nothing.
  • "Kill means we failed." Wrong, and the belief is what makes kills late and therefore catastrophic. A kill at the right gate frees capital, frees your scarcest reviewers, and removes uncompensated risk; it is a portfolio success. An organization that never kills anything is not disciplined, it is hoarding half-alive systems, each drawing supervision and each a latent incident. The fix: name a well-reasoned kill as the reviewer doing their job, separate the kill of a system from any judgment of the people who built it, and handle the real human consequences with care rather than avoiding the kill to dodge them. (see Topic 9.1)
  • "We will just keep an eye on it." "Hold steady" is almost always a build verdict too timid to admit it is spending money, or a kill verdict too attached to say the word. A deferred decision on a failing AI system is how you end up killing it four days too late, like Ofqual. The fix: force exactly one of build, buy, or kill for every system; if a system is genuinely worth holding, make it an explicit build with a small named budget and a defined next review, not a fog.
  • "The review is an accounting exercise, so finance should run it." No. The portfolio review is a governance instrument that happens to use numbers. Its purpose is to keep the kill decision yours instead of a regulator's, a market's, or a crowd's, and that is a governance judgment about risk and defensibility, not a spreadsheet. The fix: run it as a governance review with real numbers, pairing the honest return and cost with the reversibility tag, the risk cost, and the defensibility of the system, none of which a purely financial view captures.
  • "Sticker price tells us build or buy." A naive build-buy comparison pits an in-house build's engineering cost against a vendor's license fee and misses the two biggest numbers: the supervision tax on each, and the risk cost of each. A cheap-to-license vendor system you must still heavily oversee may cost more than it looks; a proud in-house build may hide a full-time reviewer. The fix: compare total cost of ownership to total cost of ownership, each with its full supervision tax (see Topic 8.2) and risk cost included, not sticker price to sticker price.
  • "A high return justifies keeping it, whatever the risk." Not for a one-way door. A system that decides who gets a loan, a place, a benefit, or a grade can produce a fine average return and still be a kill, because its downside is irreversible harm to the people it gets wrong, and there is no cheap reversal to rescue them. Ofqual's model would have "worked" on aggregate statistics while wrecking individual futures. The fix: tag every system by reversibility, and when a one-way-door system's fairness or safety numbers waver, kill it rather than betting on an average.
  • "We can decide to kill it if and when it goes wrong." Deciding to kill in the moment it goes wrong is deciding under attachment and panic, exactly when judgment is worst, which is how the kill arrives late and expensive. The fix: set a kill gate in advance, when you are cold, a specific measurable threshold (return below cost for two quarters; supervision tax above value; a fairness metric across a line) agreed by the whole team, so that when the gate is crossed the kill is the execution of a prior decision, not a fresh fight.
  • "The build-or-buy question was settled when we launched." Stale. The market catches up, the supervision tax reveals itself, and a capability's advantage erodes; a build that was right two years ago can be a buy today, and clinging to the in-house version is sunk cost in an engineer's badge. The fix: reopen the build-buy line every quarter on current numbers (see Topic 3.2), and be willing to retire your own creation when a vendor now delivers the same honest return for less.
  • "If we remove the gates, we will move faster." Removing gates does not remove the kill decisions; it guarantees every kill happens at the most expensive possible moment, in public, on someone else's timing. A gate is not a hoop, it is a cheap exit, and its whole value is that killing at a gate is affordable while killing after it is not. The fix: treat each gate as an option to exit cheaply, and remember that the cost of a kill rises every gate you let a doomed system pass.
  • "We reverted the system quickly, so no harm done." A fast reversal does not undo the harm a one-way door has already caused. Ofqual's four-day U-turn was correct and still could not restore the university places already reallocated in the hours after results day. The fix: judge a late kill not by how fast you reversed but by how much irreversible harm occurred before you did, and let that judgment push one-way-door systems to be shipped slowly and killed at the earliest wavering, not the last.
  • "The numbers were not clear enough to act on." Usually the numbers were clear and the will was absent. Ofqual could see the downgrade rate and the private-school skew before results day; MD Anderson could see the cost curve. The signal to kill is rarely missing; the courage to read it as a kill is. The fix: pre-commit to the gate so that reading the number and acting on it are the same act, and the review is where the reading happens on schedule, not the crisis.
  • "A kill is the end of the story." A kill is a decision that must itself be defensible and documented, because it will be inspected. A system killed without a recorded reason looks, to a later auditor, like a system that vanished, and a system kept without a recorded reason looks like negligence. The fix: write every verdict, especially every kill, with its forward numbers, its gate, and its human plan, into the portfolio review, so the board inspection in the capstone finds a reasoned record, not a gap. (see Topic 13.2)
  • "The kill decision ends the system." No; the kill verdict is the decision, but the system only ends when the decommission is done, and a decommission is a small project with real risks of its own. The fix: assign every kill an owner, a date, and a plan for its dependents (migrate or gracefully cut), its data (retain or delete lawfully), and its people (support through the workforce process); a kill agreed but not executed is a system still quietly running. (see Topic 9.1)
  • "This system barely costs anything, so leave it." A small direct cost hides a large opportunity cost: the scarce supervision and management attention the lingering system consumes is attention your best systems are denied. The fix: judge a keep by what the same resources could do elsewhere, not by the modest figure on the system's own row, and free the resource for a named better use when the system is not earning it.
  • "We run everywhere, so one risk number fits the system." Wrong; the same system can carry a very different risk cost in different jurisdictions, because the law that turns a harm into a liability differs. The fix: read each system's risk cost through the actual legal and reputational exposure where it runs (a high-risk load under the EU AI Act, a lighter formal cost but real civil exposure under a principles-based regime, a specific operational cost under strict content-labeling rules), rather than defaulting to a single home-country figure.

Questions people ask

What is portfolio review?
The periodic, evidence-based examination of every AI system an organization runs, judged together rather than one at a time, that assigns each system a single verdict (build, buy, or kill) on real numbers. It is the mechanism that keeps the kill decision in the organization's hands.
What is build (verdict)?
The decision to keep investing your own effort to develop or scale an AI system in-house. It is a decision to spend more, justified when the honest return clears the honest total cost, the fit is worth owning, and you have the supervision capacity to keep it safe as it grows.
What is buy (verdict)?
The decision to acquire or replace an AI capability from a vendor rather than carry it in-house, justified when a vendor's total cost of ownership is lower or the capability gives no ownership advantage. It shifts part of the cost but never removes the deployer's oversight and monitoring obligations.
What is kill (verdict)?
The decision to stop an AI system and decommission it, justified when the honest return no longer clears the honest total cost, the risk has outgrown the value, or the system cannot be made defensible at an acceptable price. A kill at the right gate is a portfolio success, not a failure.
What is total cost of ownership?
The full forward cost of keeping a system: build or future-build cost, run cost, the supervision tax, and the risk cost. Omitting any part, especially the supervision tax or risk cost, flatters the system and corrupts the verdict.

Keep going

Watch the lesson and prove it

Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.

Watch the lesson and prove it →