Scoping the deployment: what your organization actually needs versus what the demo showed
The short answer
A demo is a performance, not a proof of fitness
It shows a best case under conditions its maker chose. Babylon's demo genuinely worked and the company still collapsed. Treat every demo as an argument for buying, and scope against your own real job instead.
What you will be able to do
- Distinguish what a demo shows (a best-case performance, under conditions its maker chose) from what a deployment requires (fitness for your real job, under your real conditions), and name the gap between them.
- Define the real job a proposed AI system must do, in writing, before judging any tool: who uses it, what decision it touches, under what conditions, and what happens when it is wrong.
- Identify the three gaps that separate a demo from a deployment: the task gap, the conditions gap, and the stakes gap, plus the incentive that widens all three.
- Produce a deployment scope memo that states what is in scope, what is explicitly out of scope, and the reality tests the system must pass on your own data and conditions.
- Apply a pre-mortem question ("what would have to be true for this demo to hold up in our deployment?") to a real proposal in front of you now.
- Decide to scope a deployment down to the narrow slice where it is actually fit, and defend a smaller, safer deployment against pressure to ship the grand version the demo promised.
- Connect the scope you write to the artifacts you already hold: your systems inventory (see Topic 0.2), your 90-day priorities memo (see Topic 0.4), and the "how my model fails" record from your first build (see Topic 1.6).
- Justify, to an excited executive, why writing down the real need before evaluating the tool is the opposite of slowing the project down: it is the only way the project survives contact with a real user.
The ideas, one by one
Define the real job before you judge the tool
Who the user is, what decision it touches, under what conditions, and what happens when it is wrong, written down before the demo colors your sense of what you needed. An unwritten need gets rewritten by the demo.
The gap comes in three named shapes
The task gap (the demo did a narrower job), the conditions gap (the demo ran under conditions you will not reproduce), and the stakes gap (the demo ignored the cost of the failure that matters most). Name each and you know what to test.
The incentive widens every gap
The person showing the demo usually profits if you buy, so the demo was built to expand your belief, not to narrow it to your need. At the extreme this becomes fraud; either way, you do your own scoping.
Reality tests are pass-or-fail, on your data, catastrophic cases first
"Correctly routes these ten emergencies to a human" is a test. "Seems accurate" is not. Build by hand the tests the vendor will not, especially on the cases where being wrong is worst.
Scope by the cost of the failure, not just its frequency
A tool right ninety-nine times that sends the hundredth emergency home is lethal, not good. Aggregate accuracy hides who gets hurt. Always ask which cases are the errors and what they cost.
A scope is defined by what you refuse to ship
The out-of-scope list, with each fenced-off case routed to a human or handled another way, is what converts an oversized ambition into a bounded, defensible deployment and stops scope creep later.
Scoping down is a win, not a retreat
Ship the narrow slice that passes your tests, fence off the rest, and say so plainly. Babylon's refusal to scope down turned a possibly viable narrow product into a collapse; a smaller safe deployment beats a grand unsafe one.
Run the pre-mortem: what would have to be true for this demo to hold up here?
Imagine it shipped and failed, and surface the assumptions the demo made. Each is a test you can run or a risk you are taking knowingly. This flips you from finding reasons to believe to finding reasons it breaks.
This memo grounds the whole shipping module
The build-buy-wrap decision, the vendor interrogation, and the shipped feature are all measured against the scope you write here, and it becomes evidence in your risk classification and conformity file later.
Scoping is the fastest path to a deployment that survives
A scoping mistake is nearly free to fix on paper and ruinous to fix after a real user is harmed. Writing the scope first is not a delay; it is the only version of the project that reaches a real user and stays there.
Be wrong on paper, aggressively, now
The scope memo is the one place a failure costs only a sentence. Imagine every failure and write its fence while a fence is still free. The organizations that get hurt are not the ones that made scoping mistakes; they are the ones that made them in production instead of on the page.
A scope is a living document, not a stone tablet
Both your understanding of the job and the system's fitness change, so every reality test, update, and incident can reopen the scope in either direction. Name, for each out-of-scope case, the test that would move it in scope, and revisit the scope on a schedule and after every incident.
The incentive behind a demo is enough to justify your own scoping
You do not need to assume the vendor is lying; a demo is structurally built to widen your belief, not narrow it to your need. Assume only that a demo is an argument for buying, and do your own scoping against your own job.
Listen to it
The same lesson, talked through by two hosts as episode 17 of the podcast.
Real cases
These examples show the demo-versus-deployment gap across different domains, regions, and years. Each is attributed. Where a case is owned in depth by a later topic, it is pointer-referenced, not retold here.
Example 1: Babylon Health, United Kingdom, 2018 to 2023 (the anchor). Babylon's June 2018 demo claimed its AI matched or beat doctors on a mock qualifying exam, scoring about 81 percent against a roughly 72 percent human average (CNBC, 2018). The RCGP said the test was a revision aid, not the real exam (MobiHealthNews, 2018); an NHS oncologist documented the symptom checker missing serious conditions, including a heart-attack presentation (Undark, 2019); and the company later conceded it had not claimed real-world superiority.
In August 2023 the company, once valued near 4.2 billion United States dollars, collapsed, and its United Kingdom business sold out of administration for about 500,000 pounds (Forbes, 2023). The teachable core: the demo performed a bounded exam task, the organization needed safe open-ended triage, and no one drew the line between them until reality did, fatally.
Example 2: IBM Watson for Oncology and MD Anderson, United States, 2013 to 2018. IBM's Watson was marketed as an AI that could recommend cancer treatments, and its demonstrations were extraordinary. The University of Texas MD Anderson Cancer Center spent about 62 million United States dollars before benching the project in 2016, after an audit found the tool could not connect to the hospital's electronic health record (EHR) system and left oncologists working from outdated data (Forbes, 2017).
Later, internal IBM documents reported by the outlet STAT showed the system had produced "unsafe and incorrect" treatment recommendations, though no patients were harmed because it was not used on real patients (Healthcare Dive / STAT, 2018). Part of the cause was a task-and-conditions gap: the system was trained largely on hypothetical cases from one hospital and deployed into another with different data and vocabulary.
The demo showed a confident oncology adviser; the deployment needed one that fit a real hospital's messy records, and the gap was 62 million dollars wide.
Example 3: Google Health diabetic-retinopathy screening, Thailand, 2018 to 2020. A model that exceeded 90 percent accuracy in the lab was deployed to screen for eye disease in 11 clinics in Pathum Thani and Chiang Mai. In the field it rejected about 21 percent of nurses' images as too low-quality to read, mostly due to lighting, and slow internet made every upload a delay (Beede et al., 2020 CHI conference; TechCrunch, 2020).
No one acted in bad faith. The lab conditions simply were not the clinic conditions, and only a real-world pilot revealed it. This is the conditions gap in its purest, most honest form: a genuinely good model that was not yet scoped to the environment it had to work in.
Example 4: The AI-washing enforcement pattern (pointer). Regulators have repeatedly found companies whose AI demos radically overstated the real system, from a drive-thru voice vendor where offshore humans handled most orders, to a shopping app whose "automation" was largely people. These cases are owned in depth by later topics on interrogating the vendor and building distrust as a skill (see Topic 3.3) (see Topic 4.1). They are named here only to make one point: the incentive to widen a demo beyond reality is strong enough that, at the extreme, it becomes securities fraud, which is exactly why you scope against your own job rather than against the seller's pitch.
Example 5: Pilot purgatory, global, ongoing (established industry pattern). Beyond the famous collapses sits a quieter, larger pattern that industry analysts have repeatedly described as "pilot purgatory": corporate AI projects that demonstrate impressively in a pilot and never reach real production, or reach it and are quietly abandoned. The common thread reported across these commentaries is a scoping failure, a pilot run under favorable conditions on a narrow, tidy slice of the work, whose results do not survive the messy, high-volume, integrated reality of the full deployment. This is the conditions gap operating at the scale of whole organizations, across every region and sector, and it is why a pilot that merely reproduces the demo's favorable conditions tells you almost nothing. (Marked established as a widely reported pattern; specific proportions vary by report and are not asserted here.)
Example 6: The everyday version, any organization, now. The famous cases are famous because they were caught. The common case is quiet. A team sees a slick demo of a document-processing AI on a clean sample contract, buys it, points it at the organization's real contracts, which are half scanned images and full of odd clauses, and quietly gets wrong answers on exactly the unusual contracts that mattered most. No headline, no scandal, just a tool that demoed on the easy case and was deployed on the hard one, because nobody wrote down the real job and tested against it first. Most of the deployments you will govern are this kind, and the discipline that saves them is the same one that would have saved Babylon: define the real job, name the gaps, test the catastrophic case, and scope to what actually survives.
What the six examples share. Across a symptom checker, an oncology adviser, an eye-screening model, a set of fraud cases, a pattern of stalled corporate pilots, and a mundane document tool, the mechanism is identical: a demo performed a chosen task under chosen conditions and showed its successes, and the harm or the waste lived in the gap between that performance and the real job, real conditions, and real stakes of the deployment. None of these required a villain. Each required only that someone trust the demo without scoping the deployment. That is the failure this topic exists to make you incapable of committing.
Where people go wrong
- "The demo worked, so the system works." A demo shows a best case under chosen conditions; it is evidence of possibility, not of fitness for your job. Babylon's demo genuinely worked as a demo and the company still collapsed, because the demo's task was not the deployment's job. Treat every demo as an argument for buying, and do your own scoping against your own real conditions.
- "High accuracy on their test means it is safe for us." Accuracy on the maker's chosen data says little about your data, your users, or your catastrophic cases. Google's retinopathy model exceeded 90 percent in the lab and still rejected a fifth of real images in the field. The number you need is performance on your job, under your conditions, on the cases where being wrong is worst, and that number almost never appears in a demo.
- "Scoping the project slows it down." Scoping is the fastest path to a deployment that survives, because it prevents the far slower disaster of shipping the wrong thing and cleaning up afterward. A scoping mistake is nearly free to fix on paper and ruinous to fix after a real user is harmed. Writing the scope first is not a delay; it is the only version of the project that reaches a real user and stays there.
- "If it is right most of the time, the rare errors do not matter." It depends entirely on which cases are the errors and what they cost. A triage tool that is right ninety-nine times and sends the hundredth, a heart attack, home is not a good tool; it is a lethal one. Aggregate accuracy averages away the pattern of who gets hurt. Always scope by the cost of the failure, not just its frequency.
- "We can just deploy the whole thing and fix problems as they come up." Some problems cannot be fixed after they happen, because the harm is the problem. You do not get to un-send an emergency patient home. Scoping down and fencing off the catastrophic cases before launch is not excessive caution; it is the difference between a fixable incident and an irreversible one.
- "The vendor built it, so the vendor understands what it can and cannot do." Often the vendor knows their headline number and their happy path, and little about how their system behaves on your messy, high-stakes edge cases, because no one made them look. Assuming the builder has already scoped for your job is exactly the assumption that lets an under-scoped system ship. Your job is to make someone test your real cases, and usually that someone is you.
- "Once it is deployed for one job, using it for a related job is free." That is scope creep, and it is how a tool that was safe for a narrow, tested purpose gets silently pointed at an untested one. A system fit for common complaints is not thereby fit for emergencies. A written out-of-scope list exists precisely so that "let's just also use it for..." meets a documented decision rather than a shrug.
- "Scoping it down means the project failed." Scoping down is often the professional's hardest and best decision: ship the narrow slice that passes your reality tests, wire the rest to a human, and say so plainly. A smaller, safe deployment that does one thing well beats a grand one that does the catastrophic case badly. Babylon's refusal to scope down is what turned a possibly viable narrow product into a collapse.
- "The real need is obvious; we do not need to write it down." If the need is not written down before the demo, the demo writes it for you, quietly reshaping your sense of what you needed to match what it showed. Writing the real job first, in your own words, is what lets you see the gap at all. An unwritten scope is not a scope; it is whatever the last impressive slide put in your head.
- "A pilot in a friendly setting proves it is ready." A pilot run under favorable conditions, with motivated users and clean inputs, reproduces the demo's conditions rather than your deployment's. A real pilot has to run under your ordinary conditions, with your ordinary users, on your ordinary bad days, or it is just a longer demo. The point of a pilot is to find the conditions gap before a real user does.
- "We trust the team who chose it, so we can trust the tool." Trust in people is not evidence about a system. Capable, well-meaning teams buy under-scoped tools all the time, because the gaps are invisible from a demo and only appear when someone tests the real job. Trust has to be earned by the system, through reality tests, not inherited from the reputation of whoever liked the demo.
- "If we do not deploy it, a competitor will, so we have to move now." Speed pressure is real, but shipping an under-scoped system is not moving fast; it is moving toward an incident. The competitor who ships the unsafe version inherits the liability with it. The durable advantage goes to the organization that ships the scoped, defensible version and is still standing after the first serious failure, which is the whole arc of this module.
- "Our lawyers or the vendor's contract will cover us if it goes wrong." A contract can move some money around after a harm; it cannot un-harm the person or repair the trust. And as later topics show, the deployer who put the system in front of a user is very often the one held to account, whatever the contract says (see Topic 5.6). Scoping is not a substitute for legal terms, and legal terms are not a substitute for scoping. You need the scope because it prevents the harm the contract can only argue about afterward.
- "The demo used real-looking data, so it reflects our conditions." Real-looking is not the same as yours. A demo can run on data that looks realistic yet is cleaner, better formatted, in a different language, or from a different population than your actual inputs. The conditions gap is about your specific reality, not about whether the demo data seemed plausible. Only your own data, run through the system, tells you how it behaves on your conditions.
- "We scoped it at launch, so the scope is settled." A scope written once and never revisited becomes stale, because both your understanding of the job and the system's fitness change over time. Vendors update models, the world drifts, and users reveal needs you did not imagine. A scope you never reopen is not stable; it is simply out of date, and an out-of-date scope defends nothing when an incident review examines it.
- "We route the risky cases to a human, so those cases are handled." Routing is only a fence if the handoff itself works: someone has to receive the case fast enough, with a clear signal that it needs attention, and without being buried under everything else in their queue. An untested handoff is a second, hidden system you have not scoped, and it can fail quietly, through delay or alert fatigue, in exactly the cases where a fast response mattered most.
- "If we narrow the scope, the vendor's price should stay the same, so it is not worth narrowing." The point of narrowing scope is not to negotiate price; it is to ship only what your reality tests support so that no user is harmed by an untested use. The value of a narrow scope is a deployment that survives, not a cheaper invoice. Confusing the two is how organizations talk themselves back into the grand version to feel they got their money's worth, and then inherit its failures.
Questions people ask
- What is demo (demonstration)?
- A prepared performance of a system, showing a best case under conditions its maker chose. A demo is evidence that something is possible in the best case; it is not evidence that the system is fit for your job, your conditions, or your worst cases.
- What is deployment?
- Putting a system into real use, where a real user with a real problem depends on it under your organization's actual conditions. The distance between a demo and a deployment is the subject of this topic. More on Deployment
- What is scope (of a deployment)?
- The precise, written definition of what a deployed system will do and, just as importantly, what it will not do. A scope names the real job, the conditions, the reality tests, and the in-scope and out-of-scope boundaries.
- What is deployment scope memo?
- The artifact produced in this topic: a short document that separates the real job from what a demo showed, names the task, conditions, and stakes gaps, lists the reality tests the system must pass, and states an explicit in-scope and out-of-scope decision. It is the first artifact of the shipping module and grounds every later shipping decision.
- What is task gap?
- The difference between the narrower or cleaner task a demo performed and the messier real job the deployment must do, such as answering clean exam questions versus safely triaging a frightened patient's vague words.
Keep going
This lesson builds Buying AI well, and that page shows the roles that hire for it. Every Certified AI Governance Professional (CAIGP) lesson.
Watch the lesson and prove it
Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.
Watch the lesson and prove it →