The vendor interrogation: questions that expose what the sales deck hides

So, picture this. You are sitting in a really well-appointed conference room. Or, you know, maybe just staring at a Zoom screen these days.

Right, exactly. And you are about to sign off on a massive AI vendor contract. It is a huge moment.

Oh, absolutely. The pressure is on. Yeah, and your operations director is sitting there practically vibrating with excitement because they just love this product.

You want it deployed yesterday. Exactly. The internal sponsor who championed the project has been pushing for months.

You've seen the highly polished sales deck. You've watched the pristine demo. And honestly, it looks totally flawed.

It always does in the demo. Right. It seems like the exact technological silver bullet your organization has been waiting for.

It's going to cut operational costs, streamline workflows, boost efficiency, all the buzzwords. The dream scenario. And the slide on the screen right now literally says 95% automated.

So, all you have to do is pick up the pen and sign. But before you do that. Yes, before you do, I want to pull you out of that conference room.

We're going to drop you straight into a fast food drive-thru lane in January of 2025. Which is, frankly, the perfect place to start because that drive-thru lane represents exactly what happens when you buy the slick slide deck instead of the actual operational reality. It really does.

So, let's look at January 2025. The U.S. Securities and Exchange Commission, the SEC, brings a really highly publicized case against a company called Presto Automation. Yeah, this is a massive deal in the governance world.

Right. So, let's look at what Presto was actually selling to investors in these massive restaurant chains. Picture a sales engineer at the front of a room.

They hit play on a video clip. A very carefully selected video clip, I'm sure. Oh, completely.

It's a customer pulling up to a drive-thru. They lean out the window and they speak this messy, complicated, highly specific order. Like changing their mind halfway through.

Exactly. They want no pickles, extra sauce, a completely different sized drink. And the environment is chaotic.

There's a soda machine hissing loudly, an ambulance siren in the distance. The worst case audio scenario. Right.

And the voice AI handles it perfectly. It reads the order back flawlessly. The slide in that sales pitch claims the system is 95% automated.

And the executives in the room, they just nod. Because why wouldn't they? The deal moves forward. Yeah.

I mean, the promise of 95% automation in a high turnover, low margin business like fast food, that is economically irresistible. It fundamentally changes the profitability of the entire restaurant. It does.

But let's look behind the curtain for a second. Because behind the scenes, from June to December of 2023, there were offshore human workers. Sitting in the Philippines and India.

Exactly. And they were manually entering roughly 70% of those orders by hand on the company's advanced pilot version. 70%.

Yeah. And if you go back to the original version of the system, human intervention was required in every single instance. 100% human in the loop.

This is why this is the defining case study for this entire field of governance and vendor management. We have to do a deep dive here. Right.

Because the stakes are huge. They are. Because here is the absolute essential point about that 95% figure.

And this is what trips up even the smartest executives. The SEC did not say that the 95% figure was a forged document. Wait, really? Really.

The vendor didn't just invent a number out of thin air and print it on a slide. The number was, according to a very specific internal definition, technically true. Hold on.

Let me get this straight. They didn't lie to the SEC or the buyers. They were manually typing in 70% of the orders, but the 95% number was technically true.

Yep. How does that math possibly work? Well, it works through the magic of definitions. They simply excluded all of those offshore workers from their internal definition of the word intervention.

Oh, wow. Yeah. When they said 95% automated, what they actually meant was that 95% of the time, the restaurant's own local staff inside the physical building didn't have to walk over to the screen and intervene.

So the offshore humans who were desperately typing the orders in real time, half a world away, were just quietly left out of the math. Exactly. They essentially defined automation as not your employees.

That is wild. I mean, if a publicly traded company can obscure 70% of its human labor behind a 95% automated metric and actually successfully pitch that to sophisticated investors and enterprise buyers, we have a massive problem. A huge problem.

It means the standard ways we evaluate enterprise software are entirely broken when it comes to AI. They are completely broken. Yeah.

Because the stakes are just, well, they're fundamentally different when you are buying an AI system compared to standard deterministic software. Like a database or a CRM. Right.

With a CRM, you know what it does. But when you make the decision to buy or wrap an AI system instead of building it from scratch internally, you are taking ultimate responsibility for a complex probabilistic piece of technology that you cannot fully see inside. It's a black box.

It really is. You didn't select the training data. You didn't break it in testing.

You literally don't know where its boundaries are. Which means we need a totally new way to interrogate these vendors before we sign. And that is exactly our mission for this deep dive.

Yes. We are going to arm you with a rigorous framework. Right.

We're taking an executive education level guide on vendor interrogation. Specifically, we're going to cover the seven families of questions that surface what an AI sales deck actively leaves out. Because right now, sitting at that desk before you sign, you are in a very specific window of time.

Explain that window. Well, you are in what we call your last cheap moment. I love that phrase.

It's crucial. Before you sign the contract, the vendor desperately wants your business. Their sales team is highly incentivized to answer your questions.

Your leverage as a buyer is at its absolute maximum. After you sign. The moment the money changes hands and you begin integration, your switching costs start climbing exponentially.

The exact same questions that we're completely free to ask today in the conference room. They will sound like hostile legal accusations next month when the system is failing in production. So this framework is about utilizing that last cheap moment before knowledge gap turns into a catastrophic organizational incident.

Exactly. So let's start with the foundation of this entire problem. Right.

Let's call this first spine concept by name. The deck is edited truth, not lies. This is so important to internalize if you want to effectively question a vendor.

It really is because it is rarely about outright fraud. Right. It's about curation.

A sales deck is a precision tool optimized to do exactly one thing. Close the deal. By design, it minimizes, softens or entirely redefines anything that introduces friction into the sales cycle.

Right. And there's a term for this, right? Yeah. The industry term for this specific failure mode is AI washing.

It's the practice of describing a product as more automated, more intelligent or more AI driven than it actually is an operational reality. And I imagine that exists on a pretty wide spectrum. A very wide spectrum.

So let's walk through that. On the harmless end of the spectrum, what does AI washing look like? On the totally honest, completely legal end, you have standard market speak. So a vendor takes an old school deterministic rules engine, a piece of software that just follows basic if then logic.

Nothing fancy. Right. Nothing fancy.

But they put a modern conversational interface on it and they call it agentic or AI driven, simply because that is the vocabulary the market currently expects and more importantly, funds. So it's just marketing. It might be slightly annoying to a purist, but it isn't malicious.

Exactly. But then on the far end of the spectrum, you have literal prosecuted deception. Like the Nate app.

Yes. Nate shopping app. Look at the U.S. Department of Justice and the SEC charging the founder.

They were pitching this seamless, autonomous AI checkout system that could navigate literally any e-commerce site on the web. It sounded like magic. It did.

But federal prosecutors alleged that the checkout process was not an advanced neural network at all. It was actually being run by human workers sitting in the Philippines and Romania. Just manually typing in credit card details.

Typing in credit card details and shipping addresses while the user just stared at a loading screen on their phone. Wow. I think it's easy for a buyer to hear stories like Nate or Presto and think, OK, the market is just full of scammers.

I just need to spot the liars. But that's a dangerous oversimplification. Right.

I want to look at the middle of that spectrum because that's where most enterprise buyers actually get hurt. It's not malice. It's more like looking at a nutrition label on a box of cookies.

I like this analogy. Yeah. Think about it.

You pick up the box and it says in massive bold letters, 90% fat free. Which sounds incredibly healthy. Right.

But if that claim is based on the physical weight of the food and the food has a super high water or sugar content, most of the actual calories you're consuming could still be coming straight from fat. So they aren't lying. No.

The cookie company isn't committing fraud. They're just picking the absolute most flattering denominator to show you on the box. That is the perfect analogy.

Yeah. They're picking a flattering denominator to highlight a specific feature while hiding the actual systemic impact on your health or, in our case, your business. And the danger is that even the most reputable, honest vendors do this.

Structurally, it is just how software is sold. But when you are buying AI, that flattering definition leaves you, the buyer, holding massive amounts of unpriced risk. Because at the end of the day, when you deploy that system to your customers, you own the deployment.

Precisely. If I buy an AI chatbot to handle customer service for, say, my airline, and that chatbot hallucinates a fake refund policy. Or worse, hurls a completely unprompted insult at a high-tier loyalty customer.

Right. That customer does not care who my third-party sauce vendor was. They don't know the vendor's name.

They only care that my airline just insulted them. The liability shift is absolute. It is.

If the system fails, it is your incident. It is your brand equity that evaporates overnight. It is your customer trust that is breached.

And it is your industry regulator knocking on your door, asking for an explanation. Now, a vendor might say, well, we have a contractual remedy. We'll pay you back some of your licensing fees if we mess up.

But a financial remedy pays you after the harm has occurred. It does not unharm your customer. It does not unpublish the viral screenshot on social media.

Exactly. Which means we can't rely on the deck. And we honestly can't just rely on the vendor's brand reputation.

We have to actively look for the specific things a sales deck will reliably hide. And there are four of them. Right.

Four specific things. The conditions of the demo, the definitions sitting underneath the headline metric, the people and third parties quietly operating in the loop, and the failures. Because no deck leads with its worst outputs.

No sales engineer boots up a PowerPoint to show you the time their model confidently fabricated a racist historical fact. Never going to happen. So if the danger lies in these flattering measurements and curated conditions, we need to look exactly at how those measurements are constructed.

If the problem is the nutrition label, we need to learn how to read the fine print on the back of the box. Which brings us to our second core spine concept. Every metric rides on a definition.

Get the definition first. A headline number in a sales presentation like 95% automated or 99% accurate or 85% autonomous resolution is entirely meaningless as a piece of information until you know the mechanics of how it was calculated. So what are the mechanics? What are we looking for? You need to identify three specific elements for every single metric.

First, you need the denominator. Meaning what is the total pool of data the percentage is measured against? Exactly. Second, you need the counting rule, which is the exact rigid boundary of what qualifies as a success versus a failure.

And third, you need the excluded population. Who or what was deliberately left out of the count entirely? Right. So let's apply this practically.

Our source material gives us seven families of interrogation questions, and the first two families tackle these metrics head on. Let's look at family one, the demo to deployment gap. This is the massive chasm between the highly curated, brightly lit environment where the demo is recorded and the messy, chaotic, completely unstructured reality of your actual real world production environment.

Because a demo is just a best case scenario dressed up in the clothes of a typical case. That's a great way to put it. Let me give you an example of how that gap destroys value.

Take the case of Babylon Health. The telehealth firm in the UK. Yes.

At one point, they were valued near $4 billion. They marketed an AI symptom checker heavily, and their demos were incredibly impressive. I remember seeing those.

They looked fantastic. But the reporting and medical scrutiny that came later indicated that its real world performance never seemed to match the massive promotional claims made in those controlled demonstrations. Why not? Because in a demo, the inputs are clean.

The user describes their symptoms in clear medical adjacent terminology. But in the real world, a panicked patient is typing messy slang, misspelling words, combining totally unrelated symptoms. Right.

The gap between the demo world and the real world was catastrophic. And the company eventually collapsed into administration. And it isn't just about the data being messy in the real world, is it? Sometimes the model they demo for you is literally structurally different from the one they sell you.

Yes. And this is a crucial mechanism for executives to understand today. There was a highly publicized case recently involving a public AI benchmark leaderboard.

Okay, what happened? A company submitted a specially tuned, hyper-optimized version of their model to the leaderboard. This version was explicitly trained to excel at the conversational patterns tested by that specific benchmark. And it scored vastly higher than the base model.

But the base model was the one you could actually buy and run in your own enterprise environment. Exactly. Same model name, totally different tuning and context window restrictions.

So it results in materially different answers. Right. If you buy the model based on the leaderboard score or the highly tuned demo, you are buying a ghost.

So what's the specific interrogation question for this family? You must ask, can we run our own hardest, messiest, most unstructured historical inputs through the exact production model we will be buying before we sign this contract? Not the demo model, not the leaderboard model. The production model running on the actual compute infrastructure we are paying for. That is a phenomenal question.

Our own hardest inputs. If they say no to that, that is a massive red flag. It's a deal breaker for me.

So that covers the demo gap. Let's look at the actual math of the metric itself. This is family two, the metric definition.

This is an area where we are seeing regulators actively stepping in because the definitions are so manipulated. Look at the 2024 Texas Attorney General settlement. This was over a healthcare AI vendor.

Yes. They settled an investigation over exactly this issue. The vendor was making representations to hospitals about a critical hallucination rate in a clinical safety critical setting.

Which sounds like an important metric. It is. But that rate means absolutely nothing until you know exactly how the vendor defined the word critical and how they actually counted it.

Right. If the AI tells a doctor a patient has a common cold instead of the flu, is that a hallucination? Yes. Is it a critical hallucination? Well, who decides that? The vendor's data labelers, usually.

And if they define critical as only hallucinations that result in, say, immediate fatality, their critical hallucination rate will look incredibly low on paper. But the model might still be incredibly dangerous for everyday diagnostics. Precisely.

Let's do a deep dive right now to show how easily a metric can be dismantled. Let's do it. We're back in our conference room.

The vendor is sitting across from us and they put up a slide that proudly declares 85% autonomous ticket resolution. As an operations director, I love that number. That sounds phenomenal.

Sounds like I can rethink my entire support staffing model, cut my tier one agents, and save millions. It is entirely designed to make you think exactly that. But let's crack the metric open using those three elements we discussed.

Denominator, counting rule, and exclusions. Watch how quickly that 85% dissolves. OK, let's start with the denominator.

You stop the presentation and you ask the vendor, 85% of what? And they would smoothly say, of your customer support tickets, of course. But you have to push further. Is it 85% of all incoming support tickets across all channels? Or is it 85% of only the specific tickets that the AI chose to attempt because it had a high confidence score? Oh, that is a massive distinction.

Right. If the AI looks at a complex billing issue, decides its confidence is low, and refuses to touch it, does that count against the 85%? Often, no. They exclude it from the denominator entirely.

So let's do the math on that. Let's say you have 100 tickets. And the bot only attempts 40% of your total incoming tickets because it only handles simple password resets.

OK, so it only touches 40 tickets. And it resolves 85% of those 40 tickets. It is actually only resolving 34% of your total ticket volume.

From 85% down to 34% just by forcing them to define the denominator. Yes. That alone completely changes the return on investment calculation.

I can't fire my tier 1 support staff if the bot is only handling 34% of the volume. Not without causing a massive backlog. But let's keep going.

What about the counting rule? How do they inflate the successes? Now you ask what exactly constitutes a resolution in your system? Good question. If a customer gets frustrated by the bot going in circles, they just abandon the chat, and the bot auto closes the ticket after 24 hours of inactivity, does your system's dashboard score that as a resolved ticket? Oh, wow. If they do, they are literally counting customer rage as a successful AI interaction.

Exactly. Or even more common, if the bot realizes halfway through the conversation that it can help the customer and it successfully routes the chat to a human agent, do you score that handoff as a success in the 85% metric? Wait, because technically the bot successfully performed the routing action? Yes. Wow.

If a handoff to a human counts as an autonomous resolution, the metric is actively measuring the exact opposite of what it implies. It's inflating the automation number with human labor. You thought you were buying an autonomous agent, but you are actually just buying an incredibly expensive routing switch.

Exactly. And what's fascinating is the headline number 85% did not change. The vendor didn't lie about the math.

But its meaning completely collapsed under the weight of two structural questions about the denominator and accounting rule. Which naturally leads us to the next massive problem. If the metric is artificially inflated because the AI is handing the hard tickets off to a human, or if a human is quietly stepping in to fix the transcription, like in the Presto drive-thru case.

We desperately need to know who those people are, where they are sitting, and how much they cost. Which takes us to our third core concept. Trace the humans and the third parties.

This covers the next two families of interrogation questions. Family three is the human-in-the-loop reality. Right.

You have to understand that enterprise automation, language words like seamless, agentic, autonomous, is specifically engineered to hide people. Think about the Amazon just walk out technology that was heavily deployed in their physical grocery store. Oh, I remember this.

You scan your app at the turnstile, you walk in, grab your groceries, put them in your bag, and you just walk out. No checkout lane. It felt like pure magic.

It felt like magic because the technology narrative told you it was magic. It was framed as an incredible triumph of computer vision and AI. But extensive reporting later indicated that the system relied very heavily on a massive workforce of humans.

Often sitting in data centers in other countries. Exactly. Manually reviewing the video feeds to verify exactly what people were picking up and putting in their carts.

So the AI is doing it, plus a room full of people thousands of miles away watching me buy cereal. Yes. And I want to be very clear here.

Human-in-the-loop is not automatically a bad architectural design. Right. A system with disciplined, well-trained human review can be vastly safer, more accurate, and more reliable than a fully autonomous neural network.

But the problem arises when a system is sold as autonomous while secretly depending on people. Because your capacity planning, your cost models, and your risk models as a business are all built on the fiction of infinite software scale. Okay, let me play devil's advocate for a second.

Go for it. Let's say I'm that operations director and I really want this deal to go through. Why do we actually care if there are hidden humans in the loop as long as the output is accurate and the price is what we agreed to? It's like if I buy a subscription to a self-driving delivery service.

I think it's all AI. But I find out later that a remote human driver takes over the steering wheel on the really hard, complicated routes. Okay.

The package still arrived on time. The price was fixed. Every enterprise system has fallback.

AWS goes down. We have backups. Why is an AI fallback suddenly a deal breaker? Why should I care how the sausage is made if it tastes good? You must care because undisclosed humans are unpriced operational risks.

Walk me through that. Let's look at a fictional scenario from our source material to illustrate exactly how this breaks your business. Yeah.

Imagine a company called Northwind Kitchens. Okay. They are buying a voice AI for their drive thrusts, very similar to the Presto situation.

The governance lead at Northwind is smart. So they ask the vendor, what happens if the vendor's remote human support team, the people cleaning up the messy orders, experiences an internet outage and goes offline during the Friday night dinner rush? A completely plausible scenario. Yeah, absolutely.

The vendor admits that if the remote humans disconnect, the system would fall back to the restaurant's own staff at the physical drive-through window. Okay. So the local restaurant staff suddenly has to put the headset back on and take the orders manually, just like they used to.

Yes. But now, look at the business case that justified the purchase in the first place. The operations director wanted to buy this AI specifically so they could cut the staffing headcount at the drive-through window.

That was the ROI. But if the vendor's offshore team goes down and the volume falls back to the local window, Northwind Kitchens cannot actually cut those staff members? Exactly. They have to keep them on payroll, standing around, just in case the vendor's secret human workforce drops offline.

The entire cost-saving premise of the deal was built on a fiction. That makes total sense. If it's pure software, it scales infinitely.

If it's humans disguised as software, it has a latency limit and a capacity limit. Precisely. A disclosed human is a design choice you can plan around, negotiate SLAs for, and build redundancy for.

A hidden human breaks your operation the moment they get overwhelmed. So what's the interrogation question we must ask for Family 3? You ask. Walk me through a single complex transaction end-to-end.

List every human who touches it, where they sit, what their latency limits are, and who legally employs them. And tracing the humans leads us directly to tracing the third-party technology. This is Family 4. Model provenance and ownership.

Right. You are about to heavily depend on a model to run a core part of your business. You need to find out whose model it actually is.

Because in the current AI ecosystem, we deal heavily with something called a wrapper. Exactly. A wrapper is a product where the core capability, the actual brain of the product, is provided by an upstream foundation model that the vendor does not own.

They just wrap their user interface in some prompt engineering around an API from OpenAI, Anthropic, or Google. Which means the vendor I'm signing a contract with doesn't actually control the core technology they're selling me. Precisely.

And this is completely different from traditional software supply chains. How so? Well, if a CRM vendor uses an open source database library, that library is static. It does its job.

It doesn't change on its own. But if an AI vendor is just a wrapper, their entire cognition engine is out of their control. Oh, right.

If that upstream provider decides to deprecate the specific model version your vendor relies on, your product changes behavior overnight. It might suddenly refuse to answer questions it answered perfectly yesterday. Or if the upstream provider drastically changes their API pricing, your vendor's entire business model might collapse and they go bankrupt.

Exactly. Going back to the Presto case, the SEC specifically noted that for a period of time, the core speech recognition technology deployed in those drive-thru units was actually owned and operated by a third party, not by Presto itself. Yes.

If you don't ask, is the core model yours or a third party's, and will you explicitly name every subprocessor in the chain, you are buying a black box supply chain. And supply chains carry enormous fragility. Okay, so we've cracked the metric, we've found the hidden offshore workers, and we've mapped the third party API dependencies.

We're doing good. Once we find all these hidden dependencies, we have to start asking what happens when those dependencies inevitably fail. Because they will fail.

This brings us to the final two foundational question families before we get into the psychology of analyzing the vendor's answers. Family five is failure and incident history. This is where we look for the silent failures.

Every system fails. There's no such thing as 100% uptime or 100% accuracy in software. And especially not in probabilistic AI.

Right. If a vendor implies their system is failure-free, they are telling you one of two things. Either their monitoring is so immature that they don't even know when they are failing, or they are actively unwilling to disclose what they know.

And in AI, the most dangerous type of failure is a silent failure. Let's break down the mechanics of a silent failure. Why is it unique to AI? It goes back to how large language models function.

They are next-token prediction engines. They do not have an internal database of truth and falsehood. They probabilistically determine the most likely next word.

Because of this, a silent failure occurs when the system produces a highly confident, completely fabricated output, rather than throwing an error message. Right. A traditional software system that throws an error 404 is manageable.

It stops, your IT team sees the error in the logs, and you handle it. But a system that fabricates a confident answer fails in a way that your customers see before you do. Give me a real-world example of this silent failure.

Let's look at the Cursor AI incident in early 2025. Cursor, the developer tool company. Yes, a highly respected company.

They deployed an AI customer support agent. And this agent confidently invented a completely non-existing company policy regarding refunds or usage limits. It just made it up.

Completely made it up, and it stated to real, paying users as absolute, unbending fact. The AI didn't stutter. It didn't throw an error code.

It just lied with perfect syntax. Customers got angry. Some canceled their subscriptions based on this hallucination.

And the vendor had to do massive public damage control. Or look at the Federal Trade Commission settlement with DonotPay in 2024. Oh, the robot lawyer.

Yeah. DonotPay aggressively marketed an AI service as the world's first robot lawyer, claiming it could replace the entire legal industry for basic tasks. But the FTC alleged they hadn't actually tested whether the AI's output matched a human lawyer's quality.

Exactly. They were making massive capability claims without an established testing history behind them. So for family five, you cannot simply ask the vendor, do you test your system extensively? Why not? Because that is a process question.

Every vendor on earth will say yes to a process question. Of course we test. You have to ask for outcomes.

What's the phrasing? You say, show me your absolute worst production outputs from the last 90 days. Walk me through this specific class of unstructured input that reliably causes your system to fail silently. And if they can't? If they cannot name their own failure modes, you end the meeting.

Wow. Okay. And that brings us to family six, data handling.

This is where things get legally perilous. Data questions are where a polished, smooth talking vendor will often get visibly, physically uncomfortable in the room. Because the honest answers to data questions often lose deals.

Exactly. You must ask, what exactly happens to the proprietary data that we and our customers send you? Are you using our data to train your future models? And worse, are you passing our data to your upstream provider so they can train their models? Right. And the risk here isn't just that our competitors get our data.

What happens if the data the vendor's model was originally trained on turns out to be a legal nightmare? That is the structural risk. We saw this play out in South Korea in 2021 with a highly publicized privacy case. What happened there? A company built a conversational AI chatbot and they trained it using deeply personal private messages that users had shared on a completely different service owned by the same company.

Without telling them. Right. The national privacy regulator stepped in and found that this reuse of data lacked valid, informed consent.

They had to destroy the data set. So if you buy a product built on poison data, you might inherit that legal exposure or your product might get shut down overnight by a regulator. So what is the specific interrogation tactic here? You have to ask if the data training off switch is legally bound in the text of the contract.

As opposed to what? As opposed to just being a toggle hidden in the user settings page that they can silently update in their terms of service next month. The discomfort you see in the room when you ask these data questions, that discomfort clusters exactly where their automation story is the thinnest. OK, we have built the foundation.

We have a first six families of questions. We know what to ask about the metrics, the humans, the models, the failures and the data. But having the questions is only half the battle.

Exactly. The real executive skill is in how we interpret the incredibly slick, highly rehearsed answers we are going to receive from a professional sales engineer who does this every single day. Which brings us to our next core concept.

Classify every answer, confirm, dodge or refuse. Because when you ask these hard structural questions, you are rarely going to get a clean, simple yes or no. Sales teams are trained to answer in a register that sounds highly responsive, enthusiastic and authoritative without actually committing to anything legally binding.

Therefore, as the interrogator, you must mentally classify every material answer into one of three buckets. Let's break those buckets down so you know exactly what to listen for. First bucket, confirm.

A confirm is exactly what it sounds like, is a specific, verifiable and ideally writable commitment. Give me an example. The production model we are deploying for you is version 4.2 and we will commit to giving you 30 days written notice before any upstream API change.

OK, that's clear. Confirms are checkable. They survive the journey from the verbal conference room conversation into the written legal contract.

These confirms are the load-bearing beams that will actually support your deployment decision. Got it. Second bucket, the dodge.

A dodge is a true-sounding non-answer. It is an absolute art form in enterprise sales. How so? The salesperson might answer a much narrower question than the one you actually asked.

Right. For example, you ask, does your system use offshore human labelers in real time? And they answer. They answer, our core AI model is 100% proprietary and developed in-house.

Ah, they answered a technology question when you asked a labor question. Exactly. Or they might substitute social proof for actual technical evidence.

You ask about their hallucination rate and they respond by naming three Fortune 500 banks that use their product. We work with JP Morgan or whatever. Which tells you nothing about the hallucination rate.

Or they might use a phrase with no verifiable referent whatsoever, like our model is state-of-the-art or we use military-grade encryption. What about legal dodges? I feel like we comply with all applicable laws is a phrase you hear constantly when you asked about data privacy. It is incredibly common and it is a classic dangerous dodge.

It sounds wonderfully reassuring, but it doesn't specify which jurisdictions laws they consider applicable or how they are complying. And more importantly, AI regulations like the EU AI Act or various state laws often place heavy duties on the deployer of the system. That is you.

Exactly. Their internal compliance does not automatically shield you from liability if the system discriminates against your customers. Now, I want to clarify something.

A dodge isn't necessarily proof of malice or a scam, right? No, not at all. It's like watching a skilled politician in a televised debate. The moderator asks a difficult question about the economy and the politician smoothly pivots to the talking point they want to discuss, like education.

They didn't lie. They just refused to engage with the danger zone. But in a vendor interrogation, recognizing a dodge means you convert that verbal evasion into a mandatory written follow-up.

That's exactly right. A dodge just means the question remains unanswered. It is a placeholder.

Are there red flags we should watch out for here? Yes. The ultimate tell across all of these interactions is if a vendor simply cannot name a single weakness of their own system. If they sit there and tell you it is flawless, that it handles everything perfectly and stales infinitely, you have a massive red flag.

A seasoned operator who has actually deployed a system in the messy real world can name its weak spots in one sentence. If they can. They're either wildly inexperienced and haven't seen a break yet, or they are entirely unwilling to be honest with you.

Both are major findings that should pause the deal. Which brings us to the third bucket and our next core concept. A refusal is information, not a wall.

Because sometimes if you press hard enough, they won't dodge. They will just flat out say, we can't share that information with you. And this is where beginners in governance often fail.

They hit a refusal. They feel awkward. They feel like the meeting failed, or they get discouraged and move on.

But an expert interrogator writes it down as a located weak point. Exactly. A refusal to answer a question or to make a commitment is extremely high value information.

The vendor has just put a giant X on the map, marking exactly where the operational truth is too expensive, too fragile, or too damaging for them to share. But surely not all refusals mean we flip the table, rip up the contract and walk out of the room, right? They have proprietary secrets to protect. Of course.

The key executive skill here is distinguishing between a load-bearing question and a peripheral question. What's a peripheral question? A peripheral question might be about their internal compensation structure for their developers. If they refuse that, fine.

And a load-bearing question. Something like, whose upstream foundation model is powering this? Or who exactly is doing the human review of our data? Or will you legally warrant that 95% accuracy number you put on the slide? If they refuse those questions, the fundamental foundation of the deal is cracked. Let me challenge that on the model provenance specifically.

Shouldn't a refusal to name an upstream model mean we automatically walk away? Well, if I'm buying standard enterprise software, like a CRM or an HR platform, that vendor doesn't tell me every single open-source code library they use to build their database. They hide their libraries as proprietary architecture. Why is AI different? Why is an AI refusal load-bearing? That is a great pushback, and it comes down to the fundamental nature of the dependency.

An invisible software library and a CRM usually performs a discrete static function, like sorting a list or compressing an image. If it breaks, they patch it. But in an AI product, particularly a wrapper, the entire capability, reasoning, and reliability of the product might rest on one single upstream foundation model.

The AI is the product. If OpenAI deprecates the specific model version your vendor relies on, your product changes behavior overnight. If they refuse on a load-bearing question, you must treat it as a confirmed active risk in your board memo until proven otherwise.

You don't necessarily walk out of the room immediately, but you weigh it incredibly heavily in the risk column. That makes total sense. So we have categorized our notes.

We have our confirms, our dodges, and our refusals. But the ultimate test of all three of those tuckets comes down to what happens when you slide a piece of paper across the table and hand them a pen. This is our next core concept and the seventh and final question family.

The contract is the product, not the conversation. This is about demanding evidence in writing. There is a massive structural difference between what a vendor will cheerfully claim while drinking your coffee in a conference room and what their general counsel will legally warrant on paper when their own revenue is on the line.

We call this the warranty gap. Let's define the specific contractual mechanisms here. We need to talk about warranty, service level, audit rate, and indemnity.

How do these apply to AI? Okay, a warranty is a binding contractual commitment that a specific claim is true backed by a legal remedy if it is proven false. And a service level. A service level or SLA is a measurable performance, standard-like guaranteed uptime, or a specific accuracy rate.

Audit rate. An audit right is your explicit contractual permission to test their system on your own inputs on a regular schedule to verify performance. And finally, indemnity.

That's their promise to cover your financial losses if their system causes a harm, like a copyright infringement claim, from their training data. I want to pause on indemnity for a second. If a small AI startup offers me full indemnity, am I actually protected? Often, no.

It is worth noting that AI insurance coverage in the broader market is rapidly narrowing. Insurers are excluding hallucination risks and copyright risks, right? Exactly. So an indemnity from a startup with six months of runway might be legally binding, but practically worthless if their insurer excludes the very AI output that caused the lawsuit.

You can't squeeze blood from a stone. This is where the interrogation gets incredibly sharp. Asking, will you put that exact phrasing in the contract is the ultimate truth serum.

It really is. Remember that 85% autonomous ticket resolution claim we deconstructed earlier? The one that shrank to 34% when we looked at the denominator and the counting rule? That original 85% claim evaporates into thin air the moment you ask the vendor to warrant a 34% resolution rate with a contractual financial penalty if they miss it. Exactly.

If we connect this to the bigger picture, this goes back to the concept of your last cheat moment. Your leverage is at its absolute peak before you sign. If the vendor dodges the contract question by saying, oh, we can sort out the specific service level metrics later during the implementation phase, you have to recognize what that really means.

In enterprise software, sorted out later reliably means concede it later. Once they have your money and the integration has begun, their incentive to agree to strict financial penalties drops to zero. Okay, so we've run the interrogation.

We've cracked the metrics, traced the humans, mapped the API dependencies, categorized everything into confirms, dodges, and refusals, and pushed for the written contract. What do we actually do with this written record on Monday morning? We need to make a decision. The interrogation must produce a written record.

An interrogation you do not write down is just a conversation you will misremember under pressure from your internal sponsor. That written record resolves into one of three operational decisions. Proceed, walk, or proceed narrowed.

Perceive means everything looks good, the confirms are strong, the risks are manageable, and they sign the warranties. Walk means the load-bearing questions were refused, the risk is unpriced, and we're out. But proceed narrowed.

You say this is the most common good outcome for an enterprise AI deployment. Yes. Proceed narrowed means the vendor is credible, they are acting in good faith, but your rigorous interrogation revealed that their system is only genuinely safe and reliable in a much narrower scope than their marketing slides suggested.

So you don't kill the deal? No, you don't kill it, but you deploy it only to the specific subset of use cases that the evidence actually supports. And you write that precise boundary into the contract and into your operational monitoring. Give me an example of proceed narrowed.

If the interrogation reveals that the voice AI handles the top 15 menu items flawlessly, but hallucinates terribly on custom dietary modifications, you proceed narrowed. You only let the AI take orders for those 15 standard items, and you build a routing rule that sends any utterance containing the word allergy or custom directly to a human immediately. You scope the deployment to the evidence.

To get to that outcome, the person running the interrogation, you listening right now, needs one specific psychological skill above all others, the willingness to refuse to move on. This is the hardest habit for beginners to build. Moving on feels polite.

Keeping the meeting flowing smoothly feels productive. You don't want to seem difficult. But the clever question is entirely useless without the willingness to ask it a second, a third, and a fourth time.

You must stay on the denominator question through all of the vendor's reframes, pivots, and smiles. The answer that arrives only under pressure after three dodges is the real operational truth. And this interrogation record is not just a one-time gate that you pass through and throw in a drawer.

It is a reusable chain of logic for your entire governance program. Exactly. The claims you tagged as confirms during the interrogation become the exact test cases for your engineering evaluation suite.

You test what they promised. The dodges become the alerts you program into your production monitoring system because those are the exact areas the vendor wouldn't reassure you about. And the refusals become the named documented risks in the memo you hand to your board of directors.

So if we want to give you the single most valuable move you can make this Monday morning, what is it? What is the immediate action item? On Monday morning, I want you to take the single most impressive, highest impact metric your current or prospective AI vendor is marketing to you. The one that made your operations director so excited. Do not accept it.

Email the vendor or bring it up in your very next meeting and ask this exact sequence of questions. What is the precise denominator for this number? Who or what is excluded from the count? And will you put that specific definition along with a service level remedy in our contract? And do not move on to the next agenda item until they answer it. Doing that is how you prevent your organization from becoming the next SEC headline for AI washing.

You are stopping the sales pitch and demanding the operational reality. Let's do a quick recap of the journey we just took. We learned that sales decks are carefully edited truths, not outright lies.

We learned that you have to crack the metric definition by finding the denominator and the counting rule. You have to trace the hidden humans and the third parties hiding behind the word autonomous. You must classify every answer into confirm, dodge, or refuse.

You have to use refusals as high value maps of weak points. And finally, you have to realize that the legally binding contract, not the friendly conversation, is the actual product you are buying. Before we wrap up, I want to leave you with a new angle to ponder, something we haven't touched on yet, but which builds directly on everything we've discussed today.

What's that? We've spent this entire deep dive treating the vendor as an outside company, an external third party trying to sell you something. Right. But what happens when the vendor is your own internal IT or data science team pitching an AI build to the C-suite? Oh, wow.

That changes the dynamic completely. Internal teams use polished slide decks, flattering metrics, and highly curated best case demos too. Their careers, their promotions, and their department budgets depend on the AI project getting approved by leadership.

So ask yourself, would your own company's internal AI pitch survive this exact same seven family interrogation? Or are they hiding human-in-the-loop fallback and API dependencies from the board? That is a phenomenal, terrifying question to end on. If your internal team can't name their own denominator, you are holding the exact same unpriced risk as if you bought it from a stranger. Thank you so much for joining us on this deep dive.

Stay curious, protect your organization, and always, always ask for the denominator.