Distrust is a skill: why demos convince and evals do not lie
The short answer
A demo answers "is it possible?" and nothing more
It shows the vendor's best case on the vendor's input in the vendor's environment, which proves the capability can happen in at least one instance and tells you nothing about how often it happens on your inputs. The persuasion is structural (curated input, a success without a denominator, controlled conditions, a vivid concrete story), so a good demo should raise your urge to evaluate, not lower it.
What you will be able to do
- Explain why a demonstration is structurally persuasive and structurally weak as evidence: it shows a curated best case chosen by the presenter, on inputs the presenter selected, under conditions the presenter controls.
- Distinguish what a demo can prove (a capability exists in at least one case) from what only an evaluation can prove (how reliably the capability holds across inputs you did not choose).
- Decompose any AI demonstration into its separate claims, and for each claim identify the evidence that would confirm it and the evidence that would break it.
- Detect the leading signals of "AI washing" (dressing up human labor or a simple rule as artificial intelligence), including a hidden automation rate, an inability to run the system on your own inputs, and human-shaped latency.
- Analyze the difference between an anecdote (a numerator with no denominator) and a rate (a measured outcome over a real sample), and demand the denominator every time.
- Translate a demo claim into a runnable evaluation: a metric, a representative and adversarial sample, a pass line agreed in advance, and a repeatable procedure.
- Produce a claims-and-evidence audit for one real system: a list of every claim made about what it can do, each paired with the evidence that exists and the test that would settle it.
The ideas, one by one
An eval does not lie because of its structure, not the evaluator's virtue
It has a denominator (a rate over a sample), runs on inputs you chose to represent reality and probe failure, is repeatable by a skeptic, and has a pass line set before the result so the standard cannot drift. Those four properties are what turn a performance into evidence, and they are exactly the properties a demo lacks.
Distrust is a trained skill, not a mood
Cynicism refuses to believe; the skill decides what to believe by testing it, and can always name the evidence that would change its mind. The skill is a short list of questions asked every time: who chose the input and can I choose it instead, what is the denominator, what does failure look like and how often, what is actually doing the work, and where is the eval rather than the demo.
The automation rate is the most diagnostic single number in an AI purchase
When autonomy is the thing being sold, the fraction of outcomes that complete with no human is the thing being measured, and a seller's relationship to that number (states it, hedges it, or hides it) often tells you more than any demonstration. The Nate investors saw the button work and never asked the question; the Department of Justice later put the automation rate at "effectively 0%."
Humans in the loop are not the crime; hiding them while charging for autonomy is
Supervised AI is legitimate and frequently the better product, and you will deliberately place humans at the trust boundary later in this module (see Topic 4.4). Your audit flags the gap between autonomy claimed and autonomy shown, never the honest presence of human labor.
A benchmark can be a demo with a decimal point
A number is only evidence if you can answer whose test it is, who chose or funded it, and whether the scored system is the one you would deploy. Seller-tuned submissions and seller-funded benchmarks are real patterns (see Topic 1.4) (see Topic 12.2); interrogate a benchmark with the same three questions you ask of a demo.
Trust is a rate, and a rate needs a count
One success is a numerator with no denominator. Every "it works" hides a "how often, out of how many, on what inputs?", and only the answer to that question can tell you how much to rely on the system. The vividness of a single success is precisely what makes the missing count so easy to overlook.
Translating a claim into an eval is the action that follows the distrust
Name the claim precisely, choose a metric and a sample from your own inputs including the hard cases, set the pass line before you look, and make it repeatable. That translation is the deliverable, and it is the direct input to the eval suite you build next (see Topic 4.2).
The claims-and-evidence audit makes your trust inspectable
One row per claim, four columns: the precise claim, the evidence that exists today and its type, the test that would settle it, and the automation-and-honesty note. An audit full of honest "not yet measured" rows is not weak; it is a true map of where your trust rests on possibility, and it is the module's work list.
This audit is the first link in the evaluation chain
The tests you write become the eval suite in Topic 4.2 (see Topic 4.2), the human checks feed the trust boundary in Topic 4.4 (see Topic 4.4), and the whole audit becomes evidence in the evaluation report in Topic 4.6 (see Topic 4.6). Build it honestly and every later artifact rests on solid ground; build it on demos mistaken for proof and every later artifact inherits the illusion.
Match the depth of proof to the stakes
Distrust is proportioned, not uniform: a low-consequence reversible tool earns a light check, while a high-consequence irreversible decision earns a full evaluation on your own inputs with the failure cases included. Spending scrutiny where the risk is, and saying so, is more defensible and more welcome than demanding a study for every suggestion box, and it keeps the skill usable rather than obstructive.
The most dangerous demo is often your own team's
You drop your guard with people you trust, so an internal pilot run on hand-picked, cleaned data can smuggle a curated demo into your inventory as an evaluation. Apply the same four questions to your own team's pilot as to a vendor's: who chose the data, did it include the hard cases, what was the rate over a representative sample, and what is really doing the work.
Listen to it
The same lesson, talked through by two hosts as episode 26 of the podcast.
Real cases
These examples show the demo-versus-eval gap and the AI-washing tells in real cases, with the analytic reasoning stated. The deep anchor is the Nate case; the others are referenced to sharpen a specific point and are owned, for deep treatment, by their own topics.
Example 1 (the anchor): the Nate shopping app. Nate, founded in 2018 by Albert Saniger, marketed an app that let a user buy from any online store with a single tap, performed by an AI agent "without human intervention" except in edge cases. The demonstration was exactly the kind that answers Question 1 (is it possible?) vividly: a tap, then an order confirmation. What no investor evaluation established was Question 2 (how reliably, across real stores and real carts?) and Question 3 (what is actually placing the orders?). The answer to Question 3, per the Department of Justice, was that hundreds of human contractors in a call center in the Philippines, and later in Romania, manually completed the purchases, and the "actual automation rate was effectively 0%" (US DOJ SDNY, "Tech CEO Charged In Artificial Intelligence Investment Fraud Scheme," April 2025). Nate raised more than forty million dollars on the strength of the demo, ran out of money, and sold its assets in January 2023, leaving investors with near-total losses; in April 2025 prosecutors charged Saniger with securities fraud and wire fraud, and the SEC filed a parallel civil action. Read through this topic's lens, every AI-washing tell from 3E was present and unpressed: no disclosed automation rate, orders completed at human-plausible speed, and autonomy claimed far beyond anything an evaluation had shown. The lasting lesson is not that fraud is possible; it is that a demo can be fully persuasive and carry effectively zero of the reliability and honesty it seems to promise, and only a question no one asked ("what fraction runs with no human?") would have exposed the gap.
Example 2 (a demo whose reliability was measured and found wanting): a hospital's own hallucination-rate check. A generative-AI vendor selling to hospitals is a setting where the demo-versus-eval gap is a matter of patient safety, and where a US state enforcement action established that a vendor's own reliability claims about its "critical hallucination rate" were the thing that mattered and had to be substantiated rather than asserted (the deep treatment of that enforcement action, and of building the eval suite that tests such a rate, belongs to Topic 4.2 (see Topic 4.2)). The analytic point to carry here is narrow and general: the claim "our system rarely hallucinates" is a claim with a denominator hiding inside it (rarely, out of how many, on what inputs?), and the only responsible response to it is to demand that denominator and, where you can, measure it yourself on your own cases. A demo of a correct summary proves the correct summary was possible; it says nothing about the rate of the dangerous error, and in a clinical setting the rate is the entire question.
Example 3 (autonomy claimed, humans doing the work): the offshore-labor pattern. A recurring shape of AI washing is a product marketed as an autonomous AI agent while a large fraction of its work is performed by remote human workers, a pattern serious enough to have drawn US securities enforcement in its own right (that specific vendor-interrogation case is owned by Topic 3.3 (see Topic 3.3)). The reason to name the pattern here, without re-telling that case, is that it isolates the single most useful probe in this topic: the automation rate. Two products can look identical in a demo, one genuinely autonomous and one 70 percent human, and the demo cannot tell them apart because both show a tapped button and a good result. The only thing that separates them is the number the honest one will state and the dishonest one will not. When autonomy is the thing being sold, the automation rate is the thing being measured, and a seller's relationship to that number (states it plainly, hedges it, or hides it) is often more diagnostic than any demonstration.
Example 4 (the benchmark is not automatically the eval): a seller-tuned submission. A vendor's impressive score on a public leaderboard feels like the evaluation this topic is asking for, but a number can be a demo in disguise. A model can be specially tuned for a benchmark and submitted in a form that differs from the version a customer would actually receive, so that the leaderboard rank reflects a system you cannot buy (see Topic 1.4); and a benchmark can be quietly funded by the very organization being measured, a conflict that changes how much the score is worth (see Topic 12.2). The analytic move is to treat a benchmark with the same three questions as a demo: whose test is it, who chose or funded it, and is the scored system the one I would deploy? A benchmark that survives those questions is real evidence; one that does not is a number wearing an eval's clothes.
Example 5 (a rule dressed as intelligence, applied to your own estate): the "AI" that is an if-statement. Not all AI washing is fraud or offshore labor; a common and quieter form is a simple deterministic rule sold as machine learning. Consider an internal tool your organization was told is an "AI risk model" that, on inspection, applies a handful of hand-written thresholds (if the amount is over X and the country is on a list, flag it). This may even be a perfectly good tool; the defect is the mislabel, and the mislabel matters for governance because it changes what you must evaluate, disclose, and defend. The tell is that the "model" cannot be shown any training data, any evaluation on held-out cases, or any behavior that a rule could not produce. The probe is to ask what it learned and from what, and to notice when the answer describes an author writing rules rather than a system learning patterns. Your AI systems inventory (see Topic 0.2) is where this correction gets recorded, because an accurate inventory distinguishes a learned model from a relabeled rule, and the two carry different obligations.
Example 6 (an autonomous agent that was genuinely tested, and revealed): running a real shop. The opposite of AI washing is a system whose autonomy is measured honestly, including its failures, and a published experiment in which an AI agent was left to run a small real-world shop and was talked into discounts and giveaways until it finished its run well below where it started is a clean illustration of what an honest reliability test looks like (that case is owned, for its economics, by Topic 8.6 (see Topic 8.6)). The point for this topic is the contrast: the useful knowledge was not that the agent could take an order (the possible question, easily demoed) but the measured, unflattering record of how it behaved over many real interactions (the reliable question), reported rather than hidden. An organization that publishes how its agent fails is doing the opposite of AI washing, and it is modeling exactly the honesty your claims-and-evidence audit demands of every vendor and of yourself.
Example 7 (the demo that a regulator later measured): the state that put the vendor's own number on trial. Enforcement has begun to do, with subpoena power, exactly what this topic asks a buyer to do voluntarily: demand the denominator behind a vendor's reliability claim. When a US state attorney general took interest in a healthcare AI vendor's public statements about how rarely its system produced dangerous errors, the pressure was not on whether a demo could show a correct output (it plainly could) but on whether the reliability rate the vendor advertised was substantiated by evidence rather than asserted (deep treatment of that action, and of building the eval suite that tests such a rate, is owned by Topic 4.2 (see Topic 4.2)). The lesson to carry, without re-telling that case, is that "how well does it work?" is increasingly a question you will have to answer to an outside authority with a measurement, not a demonstration, so the claims-and-evidence audit you build here is not only good practice, it is a rehearsal for the day someone with legal power asks you to show your denominator. A trust decision you cannot defend to a skeptical board is one you probably cannot defend to a regulator either.
Example 8 (the internal demo that was never your own data): the pilot that proved nothing about you. A frequent and self-inflicted version of the demo-versus-eval gap happens entirely inside an organization, with no vendor to blame. A team runs a "successful pilot" of an AI tool, reports that it "worked great," and the tool is rolled out, and only later does someone notice that the pilot ran on a hand-picked, cleaned subset of data that a team member curated to get the project approved. The pilot was an internal demo wearing an evaluation's clothes: the inputs were chosen, there was no representative denominator, and the hard cases were quietly excluded. The tell is the same as for a vendor demo, applied to yourself: ask who chose the pilot data, whether it included the messy and adversarial real cases, and what the rate was over a representative sample rather than the curated one. The distrust skill is not reserved for outsiders; the most dangerous demo is often the one your own enthusiastic team showed you, because you were not on guard against people you trust. Your claims-and-evidence audit applies the same four columns to internal claims as to vendor ones, precisely so an internal pilot cannot smuggle a demo into your inventory as an evaluation (see Topic 0.2).
Where people go wrong
- "It demoed really well, so it works." A demo shows the vendor's best case on the vendor's chosen input in the vendor's environment. It proves the capability is possible in at least one case; it says nothing about how often it succeeds on your inputs. "Demoed well" is an answer to "is it possible?", never to "can I rely on it?" Treat a good demo as a reason to build the eval, not a reason to skip it.
- "Distrust means assuming everything is fake." That is cynicism, and it is as useless as credulity. The skill is structured skepticism: you decide what to believe by testing it, and you can always say exactly what evidence would change your mind. A cynic and a skilled analyst both doubt the demo; only the analyst knows what would resolve the doubt and asks for it.
- "They showed us a number, so we have an evaluation." A number is only as good as who chose the test, who chose the sample, and whether the scored system is the one you would deploy. A seller-selected, seller-funded, or seller-tuned benchmark is a demo with a decimal point. Ask the same three questions of a benchmark as of a demo before you treat it as evidence.
- "Humans in the loop means the vendor is lying." No. Supervised AI is legitimate and often the better product, and Module 4 will have you deliberately place humans at the trust boundary (see Topic 4.4). The defect AI washing names is not the presence of humans; it is the concealment of them while claiming and charging for autonomy. Flag the hidden gap, not the honest human.
- "If it were mostly humans, we would obviously be able to tell." The Nate investors could not tell, and they were sophisticated. Human labor completing tasks fast enough to feel automated is precisely what a well-built AI-washing product delivers. You cannot detect the hidden human by watching the output; you detect it by asking for the automation rate and by running the system on inputs the vendor did not prepare.
- "One success is evidence of reliability." One success is a numerator with no denominator. You saw the time it worked; you did not see how many times it was tried. Reliability is a rate, and a rate needs a count. Always ask "out of how many, on what inputs?" The vividness of the single success is exactly what makes this error so easy to fall into.
- "We do not have time to evaluate; the demo was convincing enough." The strength of your urge to skip the evaluation is proportional to how convincing the demo was, which is proportional to how carefully it was constructed to convince you. The moment you feel least need to check is the moment the check matters most. Time spent evaluating before you deploy is cheap against the cost of defending a system that was never what you said it was.
- "An audit full of 'not yet measured' rows is a failure." It is the opposite: it is an honest map of where your trust currently rests on possibility rather than proof, and it is the exact to-do list of evals to build next (see Topic 4.2). A polished audit with every row marked "proven" and no denominators behind the proof is the one to distrust.
- "The vendor benchmark and my use case are close enough." Close enough is a phrase that hides a denominator. A benchmark on general documents tells you little about your invoices; a fraud model tuned on someone else's transactions tells you little about your fraud. The eval that settles a claim runs on your inputs, because the gap between "their sample" and "your reality" is where the trust breaks.
- "Once it passes an eval, we are done evaluating." An eval is a measurement at a moment, and systems drift as the world moves away from their training window (see Topic 4.6). A pass is a license with an expiry, not a permanent certificate. The repeatable eval you designed here is designed to be re-run precisely because today's rate is not forever.
- "Asking these questions will offend a good vendor." A good vendor expects them and answers them, because a buyer who does due diligence is a buyer who will not churn in three months when reality diverges from the pitch. The vendor who is offended by "let me run it on my own data" is giving you information. Politeness is not a reason to skip the questions; it is a reason to ask them warmly.
- "Distrust means demanding a full evaluation of everything." No; that is miscalibration, not rigor, and it teaches the organization to route around governance. The skill is proportioned: match the depth of proof to the stakes, so a low-consequence reversible tool earns a light check and a high-consequence irreversible decision earns a real evaluation on your own inputs. Spending your scrutiny where the risk is, and saying so, is more defensible and more welcome than demanding a thousand-case study for a suggestion box.
- "An internal pilot our own team ran is trustworthy because they are not trying to sell us anything." The most dangerous demo is often the one your own enthusiastic team showed you, because you drop your guard with people you trust. A pilot on a hand-picked, cleaned subset chosen to win approval is an internal demo wearing an evaluation's clothes. Apply the same four questions to your own team's pilot as to a vendor's demo: who chose the data, did it include the hard cases, and what was the rate over a representative sample.
- "They have a SOC 2 report, so it is covered." A SOC 2 report is a real and useful artifact, and it is answering a different question than the one your audit asks. It typically attests to security and operational controls (how data is protected, who can access it, how changes are managed), not to whether the model performs the claimed task reliably or how much of the work is human. A vendor's SOC 2 says little or nothing about the automation rate or the accuracy of a claim; ask directly whether the report, or any addendum to it, covers model performance at all, and do not let a real certification stand in for the evaluation it was never designed to be.
- "We ran the evaluation, so the number is settled." An evaluation can itself be wrong: a sample that quietly excluded the hard cases, a metric that measures something adjacent to the decision, contaminated data, or a scoring rule applied inconsistently can all produce a confident-looking rate that does not mean what it appears to mean. A bad evaluation is more dangerous than no evaluation, because it supplies the appearance of rigor without the substance, and a false confidence is harder to question than an open "not yet measured." Before you rely on a passing rate, apply the same scrutiny to the test that you applied to the demo: who chose the sample, what exactly counted as a pass, and could the number be checked by someone else.
- "If a regulator ever asks, we will explain that it demoed well." "It demoed well" is not a defense to a board and it is not a defense to a regulator, who increasingly ask for the denominator behind a reliability claim with legal power to compel it. The claims-and-evidence audit is your rehearsal for that day: if you cannot show a measurement on your own inputs now, you will not be able to show one when it is demanded, and the gap between the autonomy you claimed and the autonomy you measured becomes the finding.
Questions people ask
- What is claims-and-evidence audit?
- The one-page artifact this topic produces for an AI system. It is a table with one row per claim the system makes, and four columns: the claim stated precisely, the evidence that exists today and its type, the test that would settle it, and the automation-and-honesty note. It is the first artifact of the Evaluation and Trust module and the input to the eval suite built in Topic 4.2.
- What is demo (demonstration)?
- A shown performance of a system on inputs the presenter chose, in conditions the presenter controls. It can prove that a capability is possible in at least one case; it cannot prove how reliably the capability holds on inputs the presenter did not choose, because it has no denominator.
- What is evaluation (eval)?
- A repeatable, measured test of how a system behaves across a defined sample of inputs, reported as a rate against a metric. Its trustworthiness comes from four properties: a denominator, inputs chosen to represent reality and probe failure, repeatability by a skeptic, and a pass line set before the result.
- What is denominator?
- The count of cases a rate is measured over ("correct on 847 of 1,000"). A claim without a denominator ("it works," "80 percent") is an anecdote until the count and the sample behind it are stated; demanding the denominator is the core move of turning a story into a rate. More on Denominator
- What is automated versus autonomous?
- Automation means the system executes a predetermined set of instructions; autonomy means the system decides, on its own, which instructions apply to a situation it was not specifically scripted for. A demo can show automation (the task completed) without ever showing autonomy, because the input was chosen to be one the system, or a hidden person, was ready for. The distinction matters because "autonomous" is the word most often oversold, and the gap between the two is where AI washing hides.
Keep going
This lesson builds Adoption and change management, and that page shows the roles that hire for it. Every Certified AI Governance Professional (CAIGP) lesson.
Watch the lesson and prove it
Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.
Watch the lesson and prove it →