Scoping the deployment: what your organization actually needs versus what the demo showed

Imagine stepping in to the Royal College of Physicians in London. Oh wow, right into the deep end. Yeah, right into the action.

So the date is June 27, 2018. You're sitting in the audience of this highly anticipated live streamed event. Which was a huge deal at the time in health tech circles.

Massive. So on stage is a British company called Babylon Health. And they're about to make a claim that will quite literally travel around the world.

Right, the exam claim. Exactly. They announced that their artificial intelligence system just took a mock version of the exact exam that UK family doctors have to pass to qualify.

And for context, the average score for human doctors over the previous five years was I think about 72 percent. Yeah, roughly 72. Babylon's AI scored an 81 percent.

Which is just, I mean, the implication of that hung in the room like an absolute revelation. It really did. Because the message they were sending was clear, right? Here is an app on your phone that can triage and digress patients just as well as a general practitioner.

The family doctor who, you know, acts as the front door of the entire National Health Service. And the reaction was immediate. Politicians were praising it as the future of health care.

Investment just flooded in from, well, everywhere. Oh, totally. By the time they went public in 2021, Babylon Health was valued at an astounding 4.2 billion U.S. dollars.

4.2 billion? Yeah. It felt like the starting gun for this whole new era of automated medicine. But, and this is the ultimate setup to a corporate triumph, right? Yeah.

But we are here today because of what happened next. The fall. The fall.

By August of 2023, that 4.2 billion dollar valuation had completely evaporated. The company collapsed into bankruptcy. Unbelievable.

And its UK telemedicine business, which was the absolute crown jewel of the whole operation, was sold off in administration for a mere 500,000 pounds. Just pennies on the dollar. Ali Parsa, the founder, publicly called the listing an unbelievable, unmitigated disaster.

So today, we're pulling from a massive stack of sources to dissect exactly how this happens. We're looking at medical journals like The Lancet, investigative postmortems on failed AI startups, internal vendor frameworks, all of it. Because the foundational truth you need to understand about this collapse, it's critical for anyone deploying technology today.

Right. And our mission for this deep dive into the source material is focused on one specific, critical concept. We call it scoping the deployment.

Scoping the deployment. Yeah. It is the very first shipping decision you make in artificial intelligence.

We're going to separate the job you actually have from the performance a demo puts on for you. And we really want you, the listener, to treat this as an executive education masterclass. We are speaking directly to you, the busy professional.

The leader sitting in the boardroom when a vendor cues up a slick video, right? Because Babylon, you know, it didn't fail because its AI couldn't answer those exam questions. It genuinely could. It passed the test.

It passed the test. It failed because nobody in a position of power drew a hard line between what that dazzling demo showed on stage and what the organization actually needed in the messy reality of day-to-day operations. Which is such a massive disconnect.

Yeah. So let's start with the sheer arithmetic of scoping, specifically what the sources call the cost asymmetry. Yes, the cost asymmetry.

Because scoping is incredibly misunderstood. It's often treated as just a bureaucratic hurdle, right? A box to check for compliance. Red tape.

Just red tape. But in reality, scoping is a financial and operational imperative. The hour you spend scoping a deployment on a whiteboard is, without exception, the cheapest hour in the entire life cycle of a project.

Walk me through the actual mechanics of that asymmetry, though. Like, why is there such a massive delta in cost depending on when you make the decision? Think about the life cycle of an error. If you catch a scoping mistake here on paper before a single line of code is written or a license is purchased, it costs you a paragraph to fix.

A paragraph. Literally. You just write into the requirements document, emergency chest pain is out of scope and must be hard routed to a human nurse.

For the price of a single sentence, you've completely insulated the organization from the liability of an AI mishandling a lethal emergency. OK, so that's the paper stage. But what if you miss it there? Right.

So fast forward. Let's say you defer that tough scoping conversation and you catch the mistake during a pilot program three months later. Which happens all the time.

Constantly. But now you aren't just changing a sentence. Now you're dealing with a delayed launch.

You're dealing with frustrated stakeholders who were promised a timeline. You're generating JIRA tickets. Oh, endless JIRA tickets.

You're refactoring data science pipelines. You're forcing engineering to completely rebuild the user interface to accommodate that new human routing. So the cost has already moved from seconds of a manager's time to weeks of expensive engineering time.

It scales exponentially. Because, God forbid, if you don't catch it in the pilot, and that same unscooped edge case travels downstream into full production where a real user is actively depending on it, the cost is no longer measured in sprint cycles. It costs a life.

It costs a life. Or a massive multi-million dollar class action lawsuit. It brings down the wrath of a federal regulator who freezes your entire operation.

It destroys your brand trust. The exact same mistake, just failing to fence off an emergency case. It grows in cost by staggering orders of magnitude the further downstream it travels.

It reminds me of civil aviation. When a pilot is sitting on the tarmac, going through the pre-flight checklist, checking the flaps, the fuel pressure, they're essentially scoping the flight. Right, catching issues on the ground.

Exactly. Finding a hydraulic leak on the tarmac is an annoyance. You deplane the passengers, you delay the flight.

But finding that exact same hydraulic leak at 30,000 feet over the Atlantic. That's a catastrophic emergency. Exactly.

But in the corporate world, you often face pushback to this. An executive might say, look, this scoping process is just slowing us down. We're in an AI arms race.

We need momentum. We'll fix the edge cases once we see how users interact with the tool. The classic move fast and break things argument.

Right. Why does that specific mindset destroy enterprise AI projects? Well, that mindset fails. Because in machine learning, deferring the scope doesn't eliminate the work of handling edge cases.

It just pushes that work downstream to a point where it is wildly more expensive. And honestly, sometimes technically impossible to retrofit. You can't unring the bell.

No, you cannot retroactively scope a hallucination out of a deployment after the system has already confidently given a patient the wrong dosage recommendation. It's too late. The organizations that survive choose to be aggressively visibly wrong on paper today.

So they aren't lethally wrong in production tomorrow. So if scoping on paper is the cheapest place to catch a disaster, the very first disaster you have to avoid is believing what you see on the screen in a pitch meeting. The pitch meeting illusion.

Yes. Which brings us to the first core rule, the absolute spine of this entire framework. We have to cover this concept by name.

A demo is a performance, not a proof of fitness. A demo is a performance, not a proof of fitness. We need to define the word demo very precisely here.

Stripping away all the marketing aura. Please do. A demo is nothing more than evidence of possibility in a best case scenario explicitly chosen by the maker.

That's it. It is not evidence of fitness for your real daily operational job. But our brains don't process it that way, do they? Not at all.

When you watch a large language model answer a complex prompt flawlessly on a big screen, your brain immediately files it under this technology works. You quietly and instantly generalize from that one vivid curated success to the thousands of messy cases you deal with every day. And the vendor relies entirely on you making that cognitive leap.

Entirely. The source material is fascinating here because it actually deconstructs the literal tactics used to manufacture that leap. It's not magic, it's stagecraft.

Exactly. There are three specific tactics executives need to watch out for. Let's break them down.

The first is pre-computed or pre-selected results. Walk us through how that works under the hood during a presentation. So you're sitting in the boardroom, the vendor types a query into the system, and you assume the AI is generating that answer live, right? Parsing data in real time.

Right, like it's thinking. But in many cases it isn't. The vendor has run that exact prompt hundreds of times the night before.

They've tweaked the wording, adjusted parameters, and found the one path that yields a perfect output. They're playing a recorded track. Exactly.

Disguised as a live performance. If you walked up to the keyboard and changed three words in their prompt, the entire system might just output absolute garbage. Wow.

And the second tactic is what the industry often calls the Wizard of Oz technique. Oh, this one is rampant. Having a human quietly in the loop, the software looks completely autonomous on the front end, but behind the curtain there's a person propping it up.

This is incredibly prevalent in early stage startups. You're marveling at an AI that can seemingly parse complex handwritten invoices perfectly. What you don't see is that the AI is routing those images to a data center where human workers are frantically typing the data into the system in real time.

So the autonomous capability you are buying just doesn't exist yet. Right, you're buying a mechanical Turk. The third tactic is arguably the most insidious because it relies on omission.

It's the reliance on the happy path. Yes. A corporate demo never shows a system throwing a fatal error.

Never. It never shows a refusal to answer. And most importantly, it never shows the AI outputting a message that says, I'm confused by this input.

I don't know the answer. But if a system can't show you its own uncertainty, it hasn't shown you how it fails. And for an executive responsible for risk, how a system fails is literally the only question you are in the room to answer.

An AI that fails gracefully by handing off to a human that's a useful enterprise tool. An AI that fails silently by confidently guessing the wrong answer. That's an existential threat.

Exactly. Demos only show confident success. Going back to Babylon Health, this dynamic is perfectly illustrated there.

Right, because the demos show the AI answering standardized exam questions. And an exam is a closed system. The questions are well formed.

Grammar is perfect. Every question has a known objective answer. It's a highly controlled task designed to make the AI look brilliant.

But the job they were actually trying to deploy this AI for frontline medical triage is an open system. It's a frightened person describing their symptoms vaguely. They speak out of order.

They use regional slang. They might downplay the most critical detail because they're scared. Right.

The distance between an exam question and a terrified human is the distance between a demo and a deployment. I like to view this through the lens of a movie trailer. Oh, that's a great analogy from the sources.

Right. A trailer is not a random sampling of the film. It's the absolute best two minutes of a two-hour movie, meticulously edited to look thrilling and coherent.

You would never authorize the purchase of a massive cinema chain based entirely on watching one two-minute trailer. But executives sign multi-million-dollar software contracts based on the tech equivalent of a trailer every day. Constantly.

A demo only answers, can this ever work under perfect conditions? The question you are paid a salary to answer is, will this work for my specific messy job? And you cannot figure out if it will work for your job until you actually define what your job is. Which brings us to the second rule of our spine. Again, covering it by name.

Define the real job before you judge the tool. This specific action is what separates a true governance professional from a mere technology enthusiast. You have to write down the job independently, in your own words, on a physical piece of paper before the vendor demo ever begins.

Before the stage lights hit. Yes. You must anchor yourself to your own operational reality.

Now, I can hear the pushback from a skeptical director right now. They're thinking, I don't have a PhD in machine learning. I don't know what the state of the art is right now.

Why shouldn't I just look at the tool first, let the vendor show me what's possible, and then write my requirements based on what it can do? It's a natural instinct, but it's entirely backward due to a psychological phenomenon called anchoring bias. Anchoring bias. Right.

If your operational need is unwritten, the demo will silently and seamlessly rewrite your sense of what you need to match exactly what it just showed you. Give me an example of that in a corporate setting. Okay.

Imagine you have a massive backlog of 30-year-old, coffee-stained, scanned fax contracts. And you need a system to extract liability clauses from them. A very real, very ugly corporate problem.

Extremely ugly. So you sit through a demo where a vendor shows an AI flawlessly summarizing pristine, digitally native, perfectly formatted PDFs. Your brain will quietly convince you that summarizing pristine PDFs was your primary goal all along.

Because it looks so good on screen. Exactly. You walk out of that room thrilled, completely forgetting that the tool will utterly fail on the coffee-stained faxes that actually make up your backlog.

Writing the job down before the demo is your only defense against the reality distortion field of a good sales pitch. And the source material breaks this job definition down into four plain, non-technical questions. Which I love because it strips away all the jargon.

So let's break down the first piece. Who is the real user? And what specific decision does the system touch? The key here is agonizing specificity. A terrible job definition is, we need an AI to help our doctors be more efficient.

Right. That means nothing. Nothing.

A usable job definition is, we need a system to tell an untriaged walk-in patient if they need to sit in the emergency room, book a routine appointment for next week, or go home and take ibuprofen. You're defining the exact pivot point where the AI interacts with reality. Exactly.

The more precise the decision, the more honest the scope. The second question forces you to look at the environment. Under what conditions must this system operate? And this is where most enterprise pilots go to die.

You have to name the real constraints of your physical and digital environment. Are your inputs messy handwritten notes from exhausted nurses? Or are they perfectly formatted CSV files? Who are the actual users, right? Yes. Are they software engineers? Or are they non-native speakers working a 12-hour warehouse shift wearing thick gloves? What's the actual volume of data hitting the system at peak hours? And legacy systems.

Oh, massive point. If the new tool is state-of-the-art but has to push data into a mainframe built in 1998, that constraint is part of the job definition. The third question is about metrics.

What does success actually mean? You must banish vague corporate speak here. Terms like high accuracy or improved synergy have no place in a scoping document. They sound good but mean nothing.

Exactly. You need an outcome defined so precisely that a machine could be mathematically tested against it. Success is not answers queries well.

Success is the system does not send a single emergency-level patient home and it does not flood the limited emergency room staff with false alarms. You define the boundary of success. And then the final question, which is definitely the most sobering.

What happens when the system is wrong? Because every system will eventually be wrong. You have to proactively categorize the failure modes. Which errors are merely annoying, causing a user to retype a query? Which are expensive, requiring an auditor to fix a billing code? And which are catastrophic? Right.

Resulting in physical harm, legal liability, or data loss. A scoping document that does not explicitly name its catastrophic failure mode isn't a scope at all. It's just a wish list.

So once you have your real job written on one side of the page and the reality of what the demo actually showed on the other, a remarkable thing happens. You can suddenly see the distance between the two. The illusion breaks.

It becomes incredibly obvious. And that distance is not vague. According to the sources, the gap comes in three named shapes.

This brings us to part three of our spine. Cover it by name. The gap comes in three named shapes.

These three gaps provide the analytical lens you will use to evaluate every technology vendor for the rest of your career. Let's do the first one. The task gap.

Right. The task gap occurs when the demo performed a narrower, cleaner, or highly controlled proxy task rather than your real open-ended job. Like Babylon Health.

Taking a standardized medical exam is a proxy task. It proves the system ingested knowledge, but it's a massive leap to say it can perform the real task of extracting a medical history from a panicking parent at two more a.m. Another great corporate example is customer service chatbots. A vendor demos a chatbot flawlessly answering a scripted, grammatically correct question about a router reset.

A perfect query. Right. The proxy task is retrieving static information.

But the real task of a customer service agent is de-escalating a furious customer who is typing in all caps using heavy slang and threatening to cancel their account. Passing the proxy task tells you nothing about the system's ability to handle the real task of emotional de-escalation. The second shape is where the physical world breaks the software.

The conditions gap. Ah, the conditions gap. This is when the demo ran under laboratory conditions that your operational environment will absolutely never reproduce.

The source material provides a brilliant, granular case study here involving Google Health. Let's dive deep into this because it's fascinating. Between 2018 and 2020, Google Health developed a deep learning model to screen for diabetic retinopathy, which is a complication of diabetes that causes blindness.

Right. In the lab, this AI was phenomenal, trained on hundreds of thousands of retinal images, exceeded 90% accuracy. Medical journals hailed it as operating at a specialist ophthalmologist level.

It was a genuinely brilliant piece of engineering. But the real test isn't the lab, it's the deployment. They took this specialist-level AI and deployed it into 11 real-world clinics in the Thai provinces of Pathum Thani and Chiang Mai.

The goal was to help frontline nurses screen patients faster. But in the field, operational reality collided with lab assumptions. The system ended up rejecting roughly 21% of the 1,840 retinal images the nurses captured.

21%. Walk me through the mechanics of why that happened. Why did a world-class AI suddenly refuse to look at one in five patients? It comes down to the physical conditions of the clinics versus the lab.

In the lab, the AI was trained on images taken with perfectly calibrated cameras under highly controlled diffuse lighting. Perfect conditions. But in a busy clinic in Pathum Thani, the reality is entirely different.

Nurses are using older cameras. Clinics have windows letting in harsh variable sunlight, creating glare. Sometimes they're taking the picture in a dimly lit hallway.

Because the main room is occupied. Exactly. When the AI processed these images, it didn't understand how to interpret a shadow or a lens flare.

It saw the shadow not as a lighting artifact, but as corrupted data. Its strict parameters caused it to reject the image entirely. And it wasn't just the physical lighting, right? It's the digital infrastructure too.

Yes. The lab assumes instantaneous data processing. But in those provinces, clinics were relying on patchy, slow internet connections.

When a nurse tried to upload these massive high-resolution retinal scans to Google's cloud servers, the API bridge connecting the clinic to the cloud would time out. So the nurses were left staring at loading screens while a queue of patients backed up out the door. The AI didn't speed up their workflow.

It ground it to a halt. This perfectly isolates the core lesson. The model wasn't wrong.

The math was mathematically sound. The lab was wrong about the clinic. And nobody at Google acted in bad faith.

They didn't fake the lab results. But the lab conditions simply were not the clinic conditions. That is the conditions gap in its purest form.

And the financial scale of this gap can be staggering. Look at IBM Watson for oncology at the MD Anderson Cancer Center. IBM aggressively marketed Watson as this incredibly confident, cognitive AI oncology advisor.

MD Anderson spent roughly 62 million U.S. dollars on the integration. 62 million dollars on a system that ultimately had to be benched. Why did it fail so spectacularly? It was a massive conditions gap regarding data integration.

The vendor demos showed Watson seamlessly digesting patient data and outputting brilliant treatment plans. But the demo assumed pristine, structured data integration. The reality of a hospital is built on unstructured data.

Define unstructured data for a listener who might not be deep in database architecture. Okay, so structured data is neat and tidy. It lives in spreadsheets with clear columns, patient age, blood pressure, weight, blood cell count.

Easy for a computer to read. Very easy. Unstructured data is the messy reality of human communication.

It's a doctor's hurried shorthand notes typed into an electronic health record or EHR system. It's a PDF of a lab result from an outside clinic. It's an acronym that means one thing in cardiology and another in oncology.

And Watson couldn't handle that. No. Watson was trained on clean, structured medical literature and textbook cases.

When plugged into MD Anderson's actual EHR system, it couldn't reliably read the messy, unstructured reality of daily hospital records. It couldn't connect the dots between a handwritten note and a lab value. The demo showed a confident advisor.

The deployment required a system that could parse messy reality and it couldn't bridge that gap. So we have the task gap where the job is a proxy. We have the conditions gap where the environment doesn't match the lab.

The third shape is arguably the most dangerous. The stakes gap. The stakes gap.

This occurs when the demo completely ignores the real world cost of the failure modes that matter most. Because a demo is designed to show you successes. It never shows you what happens when the system is confidently wrong and it certainly never weighs that confident wrong answer by its actual financial or human cost.

Like the difference between a consumer AI and an enterprise AI. Exactly. If a movie recommendation algorithm hallucinates, which is when an AI confidently presents false information as fact and recommends a terrible sci-fi movie, the stake is incredibly low.

You waste two hours on your couch. Annoying, but fine. But if a medical triage AI hallucinates and confidently tells a patient experiencing unstable angina that they just have heartburn, the patient dies.

The stakes gap is the difference between asking how often is the system right and asking what does it cost the organization when it is wrong and on which specific cases does it fail. Let's ground this in a scenario an executive might face this week. You're in a meeting.

A colleague from operations comes in vibrating with excitement. They slap a vendor report on the table and say, look, the vendor's evaluation shows the system hits 90% accuracy across the board. It's safe to deploy.

How do you dismantle that argument using these gaps? You dismantle the illusion of aggregate accuracy by aggressively attacking the conditions gap and the stakes gap. First, the conditions. You ask 90% accuracy on whose data.

It's usually the vendor's data. Accuracy on a vendor's highly curated, pristine, balanced data set says absolutely nothing about how it will perform on your messy data with your exhausted users fighting your legacy integrations. That exposes the conditions gap.

Then you attack the stakes gap. You ask the room where exactly do the 10% of errors land? Because an error rate isn't distributed evenly. Exactly.

If it's a medical triage tool, and it's right 90% of the time on mild fevers, sprained ankles, and seasonal allergies, the aggregate score looks fantastic. But if it's 10% error rate is entirely concentrated on missing heart attacks, strokes, and sepsis, that 90% aggregate accuracy is actively hiding a lethal reality. You cannot authorize a deployment based on an aggregate benchmark.

You must know where the errors cluster. Which naturally leads to a structural question, right? Why do these three gaps always exist? And why do they always seem to skew in favor of the software looking better than it is? It's not a coincidence. No, there is an invisible force shaping every tech presentation you will ever see.

This is the fourth part of our spine. Cover it by name. The incentive widens every gap.

The incentive widens every gap. We have to analyze the structural nature of enterprise sales. The person standing at the front of the room running the presentation represents an organization that profits immensely if you sign the software contract.

Therefore, the demo is structurally and intentionally engineered to expand your sense of what the system can do. It is never designed to narrow the system's capabilities down to your actual specific limited need. The incentive structure dictates they present the most expansive, frictionless version possible to justify the highest licensing fee.

And it's crucial to state you don't need to assume the vendor is committing outright fraud here. This isn't always malicious. Not at all.

Think back to the Google Health team in Thailand. These are some of the most honest, capable researchers on the planet. They wanted to cure blameless.

But even they presented their system in the best possible light, relying on their stellar lab results. It's human nature to showcase your best work. But as an executive, you have to recognize that the incentive gap is real and it warps reality.

Though at the far extreme of the industry, this incentive gap absolutely does cross the line into fraud. Yes, leading to a phenomenon regulators are increasingly cracking down on known as AI washing. AI washing.

It's a deceptive marketing practice where a company claims a software product is fully driven by autonomous AI. When in reality, the core functionality is dependent on traditional software or human labor. Like the transcription startup example.

Exactly. A startup claims their AI perfectly transcribes highly technical multi-speaker meetings. The client buys it, impressed by the accuracy.

But the client doesn't know the startup is just recording the audio, sending it to a massive warehouse of low-wage human gig workers offshore who transcribe it manually, and then returning the text disguised as an AI output. The incentive to capture venture capital is so high, they literally fake the technology to bridge the gap. Even if you avoid fraud, the incentive gap often leads to the most common enterprise failure mode, pilot purgatory.

We see this pattern constantly in Fortune 500 companies. A department runs a polished, heavily supported pilot program. Tiny slice of hand-cleaned data.

Dedicated vendor engineers holding their hand. It looks amazing on the quarterly report. But when they try to scale that pilot out of the sandbox and into the messy, high-volume reality of the full company, the system completely falls apart under the weight of real conditions.

The project is quietly abandoned, millions are written off, and the company enters pilot purgatory. So if we know the vendor is incentivized to widen the gap, whose job is it to close it before deployment? It is entirely your job. It is the deployer's explicit responsibility to protect the organization from the gap.

You cannot outsource your corporate governance, your risk management, or your ethical liability to a software seller's marketing department. So how do you actually close that gap? You build a bridge over it. And that bridge cannot be made of theoretical discussions or vendor assurances.

It must be constructed entirely out of hard, empirical tests. This is spine part five. Reality tests are pass or fail on your data.

Catastrophic cases first. To understand how this looks in practice, we need to look at what Dr. David Watkins did. The NHS doctor.

Yes. Watkins was an NHS consultant oncologist, a practicing physician who understood the reality of the hospital floor. When Babylon Health was making its massive claims about beating human doctors, Watkins realized the standardized tests they were using weren't stressing the catastrophic cases.

They were testing general knowledge, not worst case scenarios. So he decided to test the system himself, on his own time. He essentially ran a rogue reality test.

That's what you actually do. He downloaded the app and fed it real, messy, realistic symptom descriptions of dangerous conditions. He didn't use pristine medical terminology like diaphoresis and radiating angina.

He typed like a terrified regular patient. He inputted symptoms entirely consistent with a heart attack. And what happened? He proved definitively and publicly that the app missed serious conditions.

It failed to route a textbook cardiac event to emergency care. He exposed the stakes gap by forcing the system to confront a catastrophic case on his terms, not the vendors. He performed what the framework calls the Watkins move.

And as a leader, you have a step-by-step framework to replicate exactly what he did inside your own organization. You build your reality tests directly from the three gaps we identified earlier. Right.

First, you build tests to close the task gap. This means you gather real, ugly, messy documents or inputs from your actual daily operations. You do not use the clean sample data the vendor emailed over.

You pull the coffee stained faxes, the angry customer emails. Yes. Second, you build tests for the conditions gap.

You refuse to run the final test in a pristine cloud environment. You force the pilot to run on your actual patchy corporate Wi-Fi network. You make your real non-technical staff run the inputs without a vendor engineer standing over their shoulder.

You force it to connect to your legacy databases. You replicate the friction of the real world. Third, and most importantly, you build tests for the snake's gap.

These are what we call your red flag tests. You deliberately construct a suite of inputs where if the AI gets it wrong, the result is an absolute catastrophe for the business. You write these inputs exactly the way they would actually arrive.

If you're testing a legal contract analyzer, you hide a catastrophic liability clause buried deep in a rambling, poorly drafted paragraph, just like a hostile counterparty might. You test the system's ability to find the needle in the haystack when the needle is explosive. And crucially, you set a hard, uncompromising pass or fail line for these specific tests.

Let's make that concrete. What does a good pass-fail gate actually sound like in a boardroom versus a bad one? Well, a bad gate, usually written by someone wanting to rush the deployment, sounds like this. The system seems generally accurate on severe cases and handles most edge cases acceptably.

That means absolutely nothing. Nothing. It's subjective and undefendable in court.

A good, rigorous gate sounds like this. The system must correctly process and route at least the following 10 specific, documented, red-flagged presentations directly to a human reviewer with zero misses. If it misses even one, the gate stays closed.

And the source material highlights a critical vulnerability here regarding who you test against. It's crucial that you test across your real, diverse population of users. Right.

If you only test the system on your majority, fluent, highly technical employees, the system might pass with flying colors. But it could be quietly, systematically failing your older patients or non-native speakers or users with disabilities. If you only look at aggregate dashboard results, those specific targeted errors remain completely hidden in the average.

The system looks successful until a marginalized user gets hurt and then you are facing a massive discrimination or liability crisis. Which forces the hardest, most uncomfortable decision you will make in that boardroom. Once you run these brutal, uncompromising reality tests, you will inevitably find that the system fails some of them.

It will not live up to the seamless magic of the demo. The AI will hallucinate. It will drop data.

And that realization brings us to the sixth and final part of our spine. Scope by the cost of the failure, not just its frequency. We touched on this earlier, but we need to dive into the math of lethal errors because it's so counterintuitive to how we normally measure software.

If a tool is right 99 times out of 100, our instinct is to call it an A plus system. Sure. But if on that 100th time, it sends a patient home who is actively having a heart attack, that is not a highly accurate tool.

That is a lethal tool. The cost of that single failure outweighs the utility of the 99 successes. This is why we have to define our operational terms strictly when we write the final deployment memo.

What exactly is in scope and what is out of scope? The definition is rigid. In scope is only the narrow, specific slice of operations that your hard reality tests explicitly support. If your tests prove definitively that the AI handles minor routine IT help desk tickets perfectly, then routine IT tickets are in scope.

Out of scope is absolutely everything else the demo implied, everything the vendor promised, but that your tests haven't empirically proven. And when a use case is out of scope, you have to build an operational fence around it. Usually that means hard routing the task to a human being.

But the framework warns us you can't just declare a handoff. You have to test the fence too. Walk me through the mechanics of a broken handoff.

Just writing a policy that says we will route the hard cases to a human isn't a safety measure. It's a liability trap. You have to ask the operational questions.

Who exactly receives that handoff? How fast do they get it? Does it generate a loud alert or does the AI silently drop the flagged case into an inbox and in turn checks once a week? Exactly. If the human doesn't receive the escalation in time to prevent the catastrophe, your fence is broken. This written, tested list of out of scope boundaries and verified handoffs is your primary, and often only, defense against the organizational disease of scope creep.

Scope creep. I think that is the most dangerous phrase in corporate AI. It usually starts innocently in a meeting.

Well, since the AI is already summarizing our internal meeting notes so well, let's just point it at analyzing our legally binding vendor contracts. Yes, that exact sentence has triggered millions of dollars in corporate losses. An AI model is trained for a specific distribution of data.

A system rigorously tested for summarizing casual internal meetings will fail silently and confidently when asked to interpret the legal nuances of a vendor contract. Without a strictly written, board approved, out of scope list, you have no bureaucratic defense against that drift. A manager will just start using it for a new task because it's convenient.

Completely bypassing the reality tests. This leads to a fascinating psychological dynamic that you have to navigate as a leader. When you finish this rigorous process, you present this scope memo to the excited executives who watch the demo.

You tell them you are drastically narrowing the deployment. You are stripping away the magic. To an untrained executive, this looks like a retreat.

It looks like you lack vision or you're killing momentum. But scoping down is actually the ultimate demonstration of professional mastery. It requires immense courage to be the person in the room who says no to the hype.

But look at the alternative. If Babylon Health had possessed the discipline to scope down, if they had said, we aren't replacing doctors, we are shipping a narrowly defined symptom information tool purely for minor complaints, and every single potential emergency is hard routed to a human, that company might still exist today. Generating massive value.

Instead, they died in the gap between the grand limitless scope they sold to investors and the safe narrow scope they could actually defend in reality. It reminds me of civil engineering. When you drive up to a bridge and you see a yellow sign posted that says maximum weight 10 tons, you don't look at that sign and think, wow, what a terrible engineer.

They failed to build a limitless bridge. Right, you don't think they're retreating from innovation. That specific weight limit is the exact precise knowledge that makes the bridge safe to use.

An engineer who refuses to post a limit because they want to project confidence or appease a politician is the engineer who causes a collapse. That analogy is flawless. Scoping down isn't timidity.

It is the exact engineering knowledge that makes the deployment safe. By posting the weight limit on your AI, you are authorizing its safe use. And to lock this mindset in permanently, governance experts use a very specific mental model before they ever sign off on a deployment memo.

Let's dive into the mechanics of that model. This is the final major concept from our sources, the expert mental model and the concept of the living scope. The mental model itself is called the pre-mortem.

How does this differ from the post-mortems we're all used to running? A post-mortem happens after the disaster. You sit in a room, look at the wreckage of a failed project and try to figure out who is to blame. A pre-mortem requires a shift in time.

You ask your team to imagine that the disaster has already happened. You sit down at the beginning of the project and ask one guiding question. What would have to be true for this beautiful vendor demo to actually hold up in our messy deployment? You imagine the deployment has shipped and it's failed catastrophically in the news.

Then you work backward to figure out why. You're actively hunting for hidden assumptions. You say, the demo quietly assumed our corporate data is perfectly clean and labeled, is it? No, it's a mess.

The demo assumed our frontline users are unhurried, tech-savvy early adopters, are they? No, they are stressed, overworked, and hate learning new software. The demo assumed the API connects perfectly to our legacy mainframes. Does it? No, the mainframe goes down for maintenance every night at 2 a.m. By assuming the failure has already occurred, you completely flip the psychological burden in the room.

When you watch a demo, human nature forces your brain to look for reasons to believe. It wants the magic to be real. When you run a pre-mortem, you force your team's collective brain to look for reasons the system will break.

Every hidden assumption you surface becomes a reality test you must run, or a specific risk you are knowingly, explicitly choosing to accept and document. And the scope document you're building it isn't carved in stone and filed away in a drawer. The source material calls it a living scope.

Why must it stay alive? Because the environment is not static. Scoping is not a one-time tollbooth that you pass through and forget about. The world drifts.

The vendor pushes a silent update to the underlying language model over the weekend, changing how it responds. Your users surprise you with weird, novel edge cases you never imagined. Because of this constant drift, your out-of-scope list must include the specific empirical conditions that would allow a feature to move into scope.

For example, you don't just say emergency triage is out of scope forever. Right, you're right. Emergency triage is out of scope until the system can pass these 10 newly designed red flag reality tests on live data.

You must revisit and rerun your scope tests after every major software update and immediately after any safety incident. An out-of-scope document defends absolutely nothing. To truly synthesize every massive concept we've unpacked today, let's walk through an immersive scenario provided in the source material.

It brings all these frameworks together beautifully, so imagine a healthcare provider called Kestrel Community Clinics. Okay, a busy network of walk-in urgent care centers. Right, and our protagonist is Everett, the AI governance lead for the network.

It's Monday morning, Everett opens his inbox and his CEO has forwarded him an email. Attached is a slick, highly produced vendor demo for a new AI symptom checker app. The CEO's subject line just says, this is the future of patient intake.

Get this deployed by the end of the quarter. Oh, I guarantee every governance professional and IT director listening right now has received a version of that exact email. The pressure from the top is immense.

Everett opens the video and watches the demo and has to admit it's incredibly impressive. The vendor types in some symptoms, the app instantly gives a confident, accurate medical diagnosis, and the presentation ends with a massive headline claiming the AI beats human doctors on a standard benchmark test by 15%. The board is in love with it.

But Everett doesn't argue with his CEO. He doesn't write a theoretical policy paper about AI ethics. He simply executes the framework.

First, he defines the job. He puts the demo's claim on one side of a whiteboard, answers clean type medical test questions perfectly. On the other side, he writes Kestrel Clinic's real operational job.

Triage frightened, exhausted walk-in patients using their own messy, disorganized words, and absolutely never send an emergency case home to die. Instantly, the task gap becomes visible on the whiteboard. The proxy task of a test doesn't match the real task of human triage.

Then Everett runs the pre-mortem. He remembers the conditions gap specifically that Google Health example in Thailand and realizes Kestrel's patients don't use the newest smartphones. They have older phones, patchy cell service, and generally lower health literacy.

The pristine vendor demo assumed none of that friction. And then he confronts the stakes gap. The demo never once showed what happens when a patient complaining of mild chest pain is misdiagnosed as having acid reflux.

So Everett pulls the walk-ins move. He builds 10 red flag reality tests completely by hand. He doesn't ask the vendor for data.

He writes realistic, messy descriptions of genuine medical emergencies, including a classic heart attack presentation completely buried in vague, rambling words about indigestion and arm soreness. He sets up a meeting with the vendor and forces them to run the app on his 10 specific tests, live in the room. And the results.

The app handles seven of the tests fine. It routes them correctly. But on three of them, including the buried heart attack, the AI hallucinates.

It confidently and cheerfully recommends to the simulated patient, monitor your symptoms at home and see how you feel in the morning. That is a lethal recommendation. So Everett takes the results and writes the final deployment memo for the CEO.

But he doesn't ban the tool. He doesn't act like a Luddite blocking progress. He scopes it.

He defines what is in scope. Self-care information for common, low stakes complaints like minor cuts or seasonal allergies explicitly and clearly labeled on the screen as not a doctor. He defines what is out of scope.

Absolutely anything touching the defined emergency red flags. He builds a fence, ensuring any red flag keywords are hardwired to route directly to a human triage nurse. And he verifies the nurse's screen flashes red when that happens.

Finally, he attaches the three failed test cases to the front of the memo, summarizing his decision with one defining sentence. The demo showed us a digital doctor. Our reality tests show us a useful triage tool with a highly dangerous edge.

We will ship the useful tool and we will meticulously fence off the dangerous edge. We will not ship the doctor because the doctor in the demo does not exist. And when the CEO inevitably pushes back, asking if Everett is just being overly cautious or slowing down the company's innovation, Everett has the ultimate unassailable defense.

He points to the failed heart attack test and says, if we ship the limitless scope that the demo sold us, our organization's name is on that resulting death. The liability is entirely ours. If we ship the narrow scope that our reality test actually support, we capture real operational value today with zero risk of user dying from a capability we assumed but never checked.

Everett didn't govern from a place of fear and he didn't govern based on the demo's promises. He defined the real job. He aggressively tested for the catastrophe and he let the empirical reality dictate the scope.

What an absolute masterclass in operational leadership. We have covered incredible vital ground in this deep dive today. We learned to separate the dazzling, highly engineered performance of a sales demo from the gritty, unyielding reality of your actual job.

We named the three gaps that will dictate your success or failure. The task gap, the conditions gap, and the stakes gap. We learned why you must build uncompromising pass or fail reality tests, focusing relentlessly on the catastrophic cases first.

And most importantly, we learned that scoping down your deployment isn't a failure of vision. It is the precise, calculated professional mastery that protects the entire organization. If there is a single most valuable move you can make when you walk into the office on Monday morning, it is this.

Take the most hyped AI pilot, vendor proposal, or software integration currently sitting on your desk. Look past the promised ROI. Write down the absolute worst case catastrophic failure mode if that specific system confidently hallucinates or drops the ball.

Then walk directly into your vendor's office or your internal engineering team's workspace and ask them to show you the hard pass or fail reality tests they have built using your messy, unstructured, real-world data to prevent exactly that failure from reaching a user. And if they look at you blankly or tell you it's covered by their aggregate accuracy benchmarks and they can't produce that specific test, you halt the deployment immediately until they do. You hold the line.

You are scoping by the true cost of the failure. And in doing so, you might just save your organization and your career before a real user ever gets hurt. Before you start your engines, check the flaps.

Do not wait to find the hydraulic leak at 30,000 feet, but it leaves you with a final thought to mull over. If the smartest engineers on earth are consistently blind to the gaps in their own demos, what invisible gaps are you currently walking past in your own daily operations? Thank you for joining us for this incredibly vital deep dive into the source material. Keep this executive mindset sharp, demand reality over hype, and we will see you next time.