The vendor interrogation: questions that expose what the sales deck hides
The short answer
The deck is edited truth, not lies
A sales deck is optimized to close a deal, so it minimizes, redefines, or omits everything that would slow the deal down. Most vendors are not committing fraud; they are selecting. The interrogation exists because the selected-out information (who does the work, whose model it is, what the metric counts) is exactly what becomes your incident later.
What you will be able to do
- Explain why a sales deck systematically hides certain classes of information, and name the incentive structure that produces "AI washing" (overstating how much artificial intelligence actually does the work).
- Apply the seven-family vendor interrogation framework (demo-to-deployment gap, metric definition, human-in-the-loop reality, model provenance and ownership, failure and incident history, data handling, and evidence in writing) to any AI vendor you are considering.
- Analyze a vendor's answer and classify it as a confirm (verifiable commitment), a dodge (a true-sounding non-answer), or a refuse (a decline that is itself information).
- Distinguish a headline metric a vendor reports from the definition underneath it, and construct the follow-up question that exposes the definition.
- Produce a written vendor interrogation record: the questions asked, the answers given, what was confirmed, what was dodged, and what the vendor would and would not put in the contract.
- Connect the interrogation to your accumulating dossier: the AI systems inventory that told you this system exists, the scoping decision that said you would buy or wrap rather than build, and the eval suite and conformity file that will later have to test whatever the vendor actually delivered.
- Judge when a vendor's answers are good enough to proceed, when they demand a narrower deployment, and when they are a reason to walk away.
- Trace a single transaction end to end through a vendor's described workflow to expose the humans and third parties an "it is automated" claim conceals.
- Convert every material verbal claim into a specific written-commitment ask (warranty, service level, audit right, or indemnity) and record what the vendor will and will not sign.
The ideas, one by one
Every metric rides on a definition; get the definition first
A number without its denominator, counting rule, and excluded population is not information. The Presto "ninety-five percent automated" figure was technically true and structurally misleading because "automated" excluded the offshore humans doing the work. Never record a number until you have asked what it counts and who it excludes.
Trace the humans and the third parties
Automation language hides people and sub-vendors. For any "it's automated" claim, trace a single transaction end to end and name every person and third party who touches it. A system sold as autonomous that secretly depends on people has a hidden cost, a hidden failure point, and a risk model built on a fiction.
A refusal is information, not a wall
When a vendor declines to answer or to commit, they have marked exactly where the truth would cost them. Weigh it: walk on load-bearing refusals (whose model, who does the work, will you warrant the number) that will not close; proceed narrowed on peripheral ones. Either way, the refusal goes in the record.
The contract is the product, not the conversation
A vendor will say almost anything in a meeting and warrant far less on paper. The single most important line in your record is the gap between what they claimed and what they will put in writing as a warranty, service level, audit right, or indemnity.
Classify every answer: confirm, dodge, or refuse
The pattern across answers matters as much as any single one. Confirms are load-bearing beams. Dodges become written follow-ups and, if repeated, findings. Refuses are located weak points. The classification is what turns a conversation into an analysis.
The interrogation resolves to proceed, proceed-narrowed, or walk
The most common good outcome is proceed-narrowed: the vendor is credible but safe only in a smaller scope than you planned, so you deploy to the subset the answers support and write the boundary into the contract and your monitoring.
The record is a link in a chain, written to be used later
The confirmed claims become your eval suite's test cases, the dodged claims become your monitoring alerts, and the refusals become the risks named in your board memo. Write the record so Topics 3.4, 4.2, 5.6, and 8.6 can pull from it. An interrogation you do not write down is a conversation you will misremember under pressure.
Match interrogation depth to stakes, on purpose
A low-risk internal tool needs a lighter pass than a system that decides things about your customers. But make the depth a conscious, recorded decision, not an accident of how much time you had that day.
The interrogation is your last cheap moment
Your leverage peaks before you sign and collapses after. Every material question you fail to force now becomes a question you have to litigate after an incident, when the vendor has your money and you have the switching cost. Force the definitions and the writing while asking is still free.
Refusing to move on is the whole skill
The clever question is worth little without the willingness to ask it a second and third time. Material claims break open on the follow-up, not the first ask, which is exactly how the Presto figure would have fallen. Stay on the denominator, the transaction trace, and the written warranty through the vendor's reframes; the answer that arrives only under pressure is the true one.
You own the deployment, so you must own the knowledge
When the bought system fails, it is your incident, your customers, and your regulator, not the vendor's. A contractual remedy pays you after the harm; it does not un-harm anyone. That is why the interrogation is worth real effort even for a reputable, fairly priced vendor: reputation and price do not transfer the deployment risk back, so the knowledge has to come with it.
Listen to it
The same lesson, talked through by two hosts as episode 19 of the podcast.
Real cases
These examples show the interrogation lens applied to documented cases. Each is real and cited; the point in each is which hidden thing a good interrogation would have surfaced.
Example 1: Presto Automation, the metric that redefined "automated" (United States, 2025). The SEC's core finding is a case study in Family 2 and Family 3. Presto told investors Presto Voice "eliminat[ed] human order taking" and reported roughly ninety-five percent "automated order completion." The interrogation questions that break this: "ninety-five percent of what, and who is excluded from the count?" and "for one order, name every human who touches it and who employs them." The answers, per the SEC, would have been that "automated" excluded offshore agents in the Philippines and India who entered orders by hand (about seventy percent of orders on the more advanced pilot version from June to December 2023, and every order on the original version), and that the underlying speech-recognition model was, for a period, a third party's, not Presto's (Family 4). The number was defensible only under a definition no buyer would assume. (SEC, Presto Automation administrative proceeding, 2025.)
Example 2: Babylon Health, the demo that outran the deployment (United Kingdom, 2023). Babylon, a telehealth firm once valued near four billion US dollars, marketed an AI symptom checker whose real-world performance never matched its promotional claims; the company collapsed into administration and was sold for parts. The Family 1 question this case rewards: "can we run our own hardest cases through the system before signing, and what is the input most likely to make it perform worse than the demo?" A demo-to-deployment gap this wide is exactly what pre-signature testing on your own inputs is designed to catch. (TechCrunch, "The fall of Babylon," 2023.) (see Topic 3.1)
Example 3: The DoNotPay "robot lawyer," the capability claim with no testing behind it (United States, 2024). The FTC alleged that DoNotPay marketed an AI service as "the world's first robot lawyer" that could "replace the $200-billion-dollar legal industry," yet the company had not tested whether its output matched a human lawyer's and had not retained attorneys to check. The settlement required consumer notice and barred unsupported substitution claims. The Family 5 and Family 7 questions this case rewards: "what testing validates this capability claim, and will you warrant it in the contract?" A capability a vendor will not test and will not warrant is a capability that does not exist for your purposes. (FTC, "Operation AI Comply," 2024.) (see Topic 5.7)
Example 4: Amazon "Just Walk Out," the automation that ran on people (United States, 2024). Reporting indicated Amazon's cashierless checkout relied substantially on people reviewing video to verify purchases, and Amazon wound the system down from its stores. The Family 3 question this case rewards: "what percentage of transactions require a human, and what happens if that human workforce is unavailable?" The lesson is not that human review is wrong; it is that a system's true cost and risk model depend on knowing the humans are there. (Bloomberg, 2024.) (see Topic 0.2)
Example 5: The healthcare hallucination-rate metric (United States, 2024). A US state attorney general settled with a clinical AI vendor over how it had represented a safety-critical accuracy metric to hospitals. Without naming the vendor's specific numbers here (that treatment is owned by another topic), the transferable point is Family 2: a metric offered to a buyer in a high-stakes setting must be interrogated for its definition and measurement, because "critical hallucination rate" means nothing until you know how a critical hallucination was defined and counted. (Texas Attorney General, 2024.) (see Topic 4.2)
Example 6: The confident-wrong support bot (United States, 2025). An AI customer-support agent for a developer-tools company invented a nonexistent company policy and stated it to users as fact; customers canceled, and the vendor had to explain publicly how the system had failed. The Family 5 question this case rewards: "what class of input makes your system fail silently with a confident wrong answer rather than an error?" A vendor who cannot answer that has not characterized their own silent-failure surface, which becomes your reputational incident. (The Register, 2025.) (see Topic 3.4)
Example 7: The AI-washed checkout prosecuted as fraud (United States, 2025). Federal prosecutors and the SEC charged the founder of a shopping app whose "AI" checkout was, they alleged, run almost entirely by human workers offshore. It is the same shape as Presto: an autonomy claim with people underneath it. The transferable point is that Families 3 and 4, pressed hard and put in writing, are not paranoia; in these cases they were the exact facts a regulator later proved. (US DOJ, 2025.) (see Topic 4.1)
Example 8: The benchmark that flattered a model (2025). It was documented that a company submitted a specially tuned, conversation-optimized version of a model to a public leaderboard, where it scored well above the ordinary public weights of the same base model. No fraud is alleged; the point is narrower and directly relevant to Family 1. A demo or a benchmark score reflects a specific model version and configuration, and the version you can buy may not be the version that produced the number. The interrogation move is to require the exact production model and version in writing and to test on it, rather than trusting a figure produced under conditions you cannot reproduce. (TechCrunch, 2025.) (see Topic 1.4)
Example 9: The reused chat data behind a consumer AI (South Korea, 2021). A company built a chatbot on personal messages that users had shared for a different service, and the national privacy regulator found the reuse lacked valid consent. Without retelling the case that another topic owns, the transferable point is Family 6: a vendor's answer to "where did your training data come from and on what consent" can expose a legal problem you would inherit as the deployer. Ask it, and ask whether the vendor will indemnify you if the answer turns out to be wrong. (PIPC, the Personal Information Protection Commission, South Korea, 2021.) (see Topic 2.2)
Cross-example pattern. Read the nine examples together and one shape repeats: the claim is real, the definition or configuration behind it is favorable, and the unfavorable part (the humans, the third-party model, the demo conditions, the field misses) is the part left off the slide. This is why the interrogation is organized by families rather than by a list of clever questions. The families map onto the categories of hidden information, so working them in order guarantees you look everywhere the shape tends to hide. The alternative is asking three sharp questions about the one thing that happened to catch your eye, and missing the family that will actually produce your incident.
Where people go wrong
- "A confident, polished vendor is a trustworthy vendor." Polish is a sales skill, not evidence. The Presto and Nate cases involved companies confident enough to make their claims to investors and regulators. Confidence tells you about the salesperson, not the system. Interrogate the claim, not the delivery.
- "The headline metric is the fact I need." The headline metric is the fact the vendor chose to give you. A number without its denominator, counting rule, and excluded population is not information. "Ninety-five percent automated" collapsed the moment the SEC asked what "automated" excluded. Always get the definition before you record the number.
- "Human-in-the-loop means the vendor is being cautious and safe." Sometimes. But a system sold as autonomous that secretly depends on people is a hidden cost and a hidden single point of failure, not a safety feature. The problem is not the humans; it is the humans being undisclosed, so your cost model, capacity plan, and risk model are all built on a fiction.
- "If I ask hard questions, I will offend the vendor and lose the deal." A vendor worth signing expects a real buyer to interrogate the product; the good ones respect it. If hard, fair questions genuinely end the relationship, the vendor was protecting something, and you just learned what. The interrogation filters for exactly the vendors you want.
- "A refusal to answer means I should walk away." Not necessarily. A refusal is high-value information about where the truth is expensive to the vendor, but the right response is to weigh it: is it a load-bearing question (whose model, who does the work, will you warrant the number) or a peripheral one? Walk on load-bearing refusals that will not close; proceed narrowed on peripheral ones.
- "The verbal assurance in the meeting is what I'm buying." You are buying the contract. A vendor will say almost anything in a room and warrant far less on paper. The gap between the claim and the warranty is the real product. Convert every material claim into a "will you put that in writing" and record the answer.
- "Once I've interrogated the vendor, governance of this system is handled." The interrogation is the first link, not the whole chain. The confirmed claims still have to be tested by your eval suite (see Topic 4.2), the dodged claims still have to be watched by your monitoring (see Topic 3.4), and the refusals still have to be managed as named risks in your board memo (see Topic 8.6). The record exists so the chain can use it.
- "AI washing only happens at scam companies." It runs on a spectrum. Outright fraud sits at one end, but honest vendors also report genuine metrics under flattering definitions and describe systems in the most autonomous language the truth will bear. The interrogation is needed for the honest vendors too, because their softening is legal and still leaves you holding risk you did not price.
- "I should interrogate every vendor with the same intensity." No. Match the depth to the stakes. A low-risk internal productivity tool needs a lighter pass than a system that makes decisions about your customers or feeds a regulated process. But make the depth a conscious, recorded decision, not an accident of how much time you had.
- "The model in the demo is the model I will get." Not necessarily. A demo can run on a different model, a different version, or a more favorable configuration than the one your contract delivers, and the deck will not flag the difference. The same model name can produce materially different answers under different tuning and context. Require the exact production model, version, and configuration in writing, and run your pre-signature test on that, not on whatever the demo used.
- "A big-name upstream provider means the dependency is safe." A reputable upstream model is more reliable than an obscure one, but it does not remove the supply-chain risk. The upstream provider can still change the model, deprecate a version, change pricing, alter usage terms, or restrict access, and any of those flows straight through to your service because your product is a wrapper around theirs. Reputation reduces some failure probabilities; it does not transfer the dependency risk off your books. Name and warrant the upstream model regardless of the provider's size.
- "The vendor told me they comply with all applicable laws, so that box is checked." "We comply with applicable law" is a dodge, not a finding, because it does not say which laws, in which of your jurisdictions, or how. It also answers the wrong question. AI-specific rules increasingly place duties on the deployer, not just the provider, so a vendor's own legal compliance does not automatically cover the obligations that land on your organization once you put the system into production. Ask what the vendor's compliance claim actually covers, and separately confirm what it leaves for you to handle.
- "An indemnity clause means I am covered if the system causes harm." An indemnity is only as good as the vendor's ability and insurance to pay it, and AI-specific insurance coverage has been narrowing, with some insurers moving to exclude AI-output liabilities. A generous indemnity from a thinly capitalized vendor whose own policy excludes the exact harm is a comfort on paper and nothing behind it. Ask what the vendor's insurance actually covers, not just what the contract promises. (see Topic 8.3)
Questions people ask
- What is vendor interrogation?
- The structured process of questioning an AI vendor, before buying, to surface the information a sales deck omits: the conditions of the demo, the definition under each metric, the humans and third parties in the loop, the failure history, the data handling, and what the vendor will commit to in writing.
- What is sales deck?
- A vendor's presentation optimized to close a deal. It is not usually false, but it is selective: it shows the best-case demo, the most favorable metric, and the most autonomous-sounding description of the system, minimizing or omitting whatever would slow the deal down.
- What is AI washing?
- Describing a product as more automated, more intelligent, or more AI-driven than it actually is. It runs on a spectrum from calling a rules engine "AI" to meet market expectations, through reporting a genuine metric under a flattering definition, to outright deception that regulators prosecute (as in the Presto Automation case, SEC, 2025).
- What is metric definition?
- The denominator, counting rule, and excluded population that give a headline number its actual meaning. "Ninety-five percent automated" is meaningless until you know ninety-five percent of what, what counts as a success, and who is left out of the count.
- What is denominator?
- The base a percentage is measured against. Changing the denominator changes what a metric means without changing the number. A central target of any metric-definition question. More on Denominator
Keep going
This lesson builds AI vendor due diligence and third-party risk, and that page shows the roles that hire for it. Every Certified AI Governance Professional (CAIGP) lesson.
Watch the lesson and prove it
Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.
Watch the lesson and prove it →