Reproducing a claim: testing a vendor benchmark yourself in an afternoon

You know, usually when we talk about a business metric, there is this expectation of, well, intrinsic reality. Right. Like it's a physical thing.

Exactly. Like accounting. I mean, you look at a balance sheet, you see the revenue line, and you can point to the bank accounts and say, there it is.

Yeah, it's verifiable. It's a hard, indisputable fact. You might argue over how to allocate the funds, but the number itself is grounded in a shared, universal reality.

Which makes sense. And when you look at traditional enterprise software, for example, uptime is uptime. Right.

99.9% means the server was online. It's a binary state measured against, you know, objective time. Exactly.

It's clean. Right. But then, you step into the world of artificial intelligence procurement right now, and you are staring at a vendor's slide deck.

Oh, yeah. A whole different world. They show you a benchmark score, they say, our new model scores 90% on logical reasoning.

Okay. And suddenly, that accounting precision just completely evaporates. It really does.

We are looking at a landscape that is honestly incredibly murky, where numbers are treated as immutable facts, but they are actually something else entirely. Which is exactly how a room full of brilliant executives ended up being totally misled by the frontier math benchmark last year. Oh, this is such a wild story.

It is. Let's look at exactly what happened, because it perfectly illustrates why taking a vendor claim on faith is, well, it's essentially trusting them to grade their own homework. So, on December 20th, 2024, OpenAI unveiled a new reasoning model called O3, and the entire industry announcement was carried by one staggering number.

The headline number. Exactly. The headline was that O3 scored about 25% on frontier math.

And we should define frontier math immediately for you, the listener, because this wasn't just any generic test. No, not at all. This is a benchmark built by an organization called Epoch AI.

Oh, yeah. It consists of incredibly hard, unpublished, research-level mathematics problems. We are talking about problems that earlier frontier models could barely touch.

They were solving maybe 2% of them on a good day. Right. And Epoch AI actually commissioned working professional mathematicians to write these problems.

They were paying them hundreds or sometimes thousands of dollars each. A massive investment. Yeah.

And the entire point of that massive expense was to ensure these questions would be genuinely, completely brand new. And that is the crucial detail here. The problems were unpublished.

Right. They were designed specifically to be too new for any model to have seen them or, you know, memorized them from the open web during its training phase. So when that 25% number dropped, the framing was unambiguous.

Exactly. The narrative was that a machine had just leapt from near-zero capability to solving a quarter of a set of problems that professional mathematicians had written to be uniquely difficult. And that headline traveled across the industry faster than any caveat could possibly follow.

Oh, absolutely. It was treated as a fact. A 25% capability in frontier mathematical reasoning.

But then the fallout happened because a 15% drop on a shipped product isn't just a rounding error. No, it's a completely different reality. Right.

And that reality surfaced only after the headline had done its work. On the exact same day as the O3 reveal, Epoch AI updated its frontier math paper. They quietly added a disclosure.

Yeah. They added a disclosure that OpenAI had actually funded the benchmark's creation. Which is huge.

And not only that, OpenAI had access to most of the problems and their solutions. Which is a fact that contributing mathematicians were entirely unaware of. Right.

They had no idea. And this led to severe public criticism. By January 19, 2025, TechCrunch was reporting on the backlash from these mathematicians.

Some of them actually stated they might not have contributed their intellectual property had they known the funding source. Wow. Now, Epoch AI stated that OpenAI had a verbal agreement not to train on the problem set.

A verbal agreement. Yeah. And they also mentioned that a separate holdout set of problems, which OpenAI supposedly could not see, was reserved for independent checking.

But the punchline here is, what happened when that independent check actually occurred? This is the kicker. When Epoch AI ran its own independent evaluation in April 2025, on the shipped default version of O3, meaning the actual version a paying customer would use, not a special lab version, the score was not 25%. It's about 10%.

Same benchmark. Same model name. Less than half the headline.

That drop from 25% to 10% is the anchor for our entire discussion today. And it brings us to the most important realization a professional can have in this space. Which is our mission for this Deep Dive.

Exactly. A benchmark score is a claim, not a fact. But I have to push back here, because, well, how is that not just outright fraud? I know it sounds like it.

We just saw a score drop from 25% to 10% with the exact same model. If a traditional software vendor claimed their database processed 25,000 transactions a second, and you bought it, and it only processed 10,000, you'd sue them. Oh, 100%.

So how is this mathematically possible without someone lying? Because a benchmark score is not an intrinsic property like mass or volume or database throughput. A benchmark score is a compressed sentence built on a highly specific series of structural choices. A compressed sentence.

Yes. Change those choices, and the number changes drastically. With absolutely no change to the underlying model itself.

And nobody in the frontier math story needed to lie for the public to end up entirely misled. Okay, so if it's a compressed sentence, let's decompress it. What are the actual mechanics of that sentence? We can break it down into four parts.

Yeah, because if we bring an unverified benchmark number into a massive procurement decision, we are risking our organization's resources on an illusion. What are the load-bearing parts of a benchmark claim that we need to be looking for? So there are four fundamental parts to that compressed sentence. The first is, what was actually measured? Right.

When a vendor says 25%, the human brain immediately generalizes that to mean the model is 25% good at general math. Which it isn't. Exactly.

And executives often generalize it further to mean it is highly capable at their organization's specific analytical tasks. It means neither of those things. It means it solved 25% of a highly specific curated set of problems.

You have to ask, what exactly is the task inside this benchmark described in plain non-marketing words? Right. Because the label on the box is not the actual task. If a benchmark is called advanced corporate reasoning, but the actual task is just extracting dates from impeccably formatted legal summaries, that's a massive difference.

It's completely different. The second part of that compressed sentence has to be the denominator, the out of what. Exactly.

Every percentage has a denominator and that denominator is a choice made by the person running the test. Right. Was the score calculated out of the full benchmark or just a curated subset that happened to look favorable? Was it the public set or was it an independent holdout set? And we should define a holdout set because it's a term that gets thrown around a lot but is rarely implemented correctly.

Very true. A true holdout set is a portion of the benchmark deliberately kept private and completely unseen by the model builder. It is reserved specifically for independent checking, acting as an empirical firewall against the model just memorizing the test.

Precisely. A model can score incredibly high on the public portion of a benchmark and much lower on a heldout portion it could not have seen. The delta between those two numbers is incredibly informative.

Oh, absolutely. But the denominator question also extends to the specific model build being tested. Right.

The source material references an incident with Elmerina in 2025. Elmerina is a widely respected public leaderboard. A specially tuned build of a model was submitted to the arena and ranked very highly.

Yeah. But when developers downloaded the base weights of the model with the exact same name to run locally, the results were totally different. Because tuning a submission fundamentally alters its behavior compared to the base weight.

The vendor was essentially submitting a race car to the track, but selling a commuter sedan to the public under the same badge. That is exactly what happened. So which exact model version is just as much a denominator question as which exact test items? Makes sense.

The third load-bearing part of the sentence is, under which setting? The setting. And this is honestly where the most sophisticated inflation hides in plain sight. Because the settings sound technical and boring, so non-technical decision makers just gloss over them.

Give us an example of how a setting inflates a score. Compute scaling is the most common. Was it run using a normal compute budget, or was it a high compute preview? Right.

Because that takes time and money. Exactly. To get some context, modern AI models can spend extra processing time, extra compute, to basically think before they answer.

The 0.325% headline, for instance, was achieved using a high compute preview configuration. Which most people won't use. Right.

That setting costs significantly more per query, and takes much longer to generate an answer. It is a setting most ordinary enterprise users would never pay for in production, because it destroys the margins of whatever product they are building. So a number is only comparable to your world if the compute setting matches your world.

And the fourth part of the decompressed sentence. By whom? Who built the benchmark? Who funded it? Who had access to its contents? And who is reporting the score? A vendor reporting its own score on a benchmark it funded and could see inside is exactly the situation that demands independent reproduction. Because the incentives are so skewed.

Every incentive in the room points toward the flattering number. To make this incredibly concrete for you, the listener, think about comparing AI benchmarks like looking at a car's MPG rating. Oh, that's a great analogy.

It is as if a manufacturer claims a new heavy-duty truck gets 50 miles to the gallon. That is the headline on the slide. Right.

But when you decompress the sentence and look at those four load-bearing parts, you realize they tested it driving downhill, the massive tailwind, the manufacturer owned the test track, and they got to choose the exact driver. That is a perfect analogy. The manufacturer didn't commit fraud.

The truck really did travel 50 miles on one gallon of gas under those precise conditions. But it's not the reality you experience on your morning commute in stop-and-go traffic with a fully loaded trailer. Exactly.

If you budget your supply chain's fuel costs based on that 50 MPG headline, your logistics company will literally go bankrupt. And if we agree the manufacturer isn't committing fraud with that 50 MPG claim, then where exactly is the inflation coming from? Right. There have to be systemic blind spots we are missing.

And there are. There are four specific structural ways a number gets inflated by those choices, even when every vendor in the room is acting with total honesty. The most pervasive culprit has to be contamination.

We touched on this with the holdout sets, but let's explain the actual mechanism here. How does a model accidentally cheat on a test? Contamination happens because of the sheer scale of how these models are built. They're scraping everything.

Exactly. They are trained by scraping vast swaths of the open web. If any of a benchmark's questions, or even close paraphrases of them, appeared anywhere in the model's training data, the model can score high by memory rather than by genuine capability.

And the open web is incredibly vast. So this isn't a vendor maliciously feeding an answer key into the server room at midnight. It's often entirely accidental.

Exactly. The sources highlight a classic example. MMLU.

Right. I've seen that one everywhere. MMLU stands for Massive Multitask Language Understanding.

It's a benchmark introduced by Hendrix and colleagues back in 2021, covering everything from high school physics to professional law. So it's very broad. Very.

And because it became the gold standard benchmark, it was discussed everywhere. Researchers posted the questions on forums. Students debated the answers on Reddit.

Developers put subsets of it into open source repositories on GitHub. And because the web scrapers vacuum up Reddit and GitHub to train the next generation of models, the benchmark slowly seeped into the training data. Precisely.

The established documented finding is that as a benchmark ages and spreads across the internet, model scores on it tend to naturally drift upward. Which is wild. It really is.

And it has absolutely nothing to do with improved underlying reasoning ability. The models just memorize the topography of the text. Which leads us to the second structural failure mode.

A flattering configuration. Yes. We mentioned compute earlier, but this goes way beyond just processing power, doesn't it? It does.

A flattering configuration refers to any setting you will never deploy, but which makes the model look brilliant in a lab. The vendor reports the score under the absolute optimal conditions. For instance, they might use highly specialized prompting tools, or agentic loops.

Explain what an agentic loop is, because that sounds like a feature, not a bug. An agentic loop is when you don't just ask the model a question and get an answer. You give the model a workspace.

A workspace, like an environment. Right. You ask it to write code, execute that code, read the error message, realize it made a mistake, and rewrite the code, iterating over and over before finally showing you the result.

Wow. Okay. The sources cite SWE Bench from Jimenez and colleagues in 2023 and 2024.

SWE Bench measures a model's ability to resolve real software engineering issues from GitHub. Scores on SWE Bench vary wildly, sometimes by massive margins, depending entirely on the agent loop and retrieval scaffolding wrapped around the model. But if I am buying an AI for a customer service chatbot, it doesn't get to write code, test a response, fail, and rewrite it for 20 minutes while the customer waits.

Exactly. It gets one shot to generate a text reply. So the configuration flatters the model, but betrays the deployment reality.

Precisely. The third failure mode is Conflict of Interest in Provenance, which we saw vividly in the Frontier Math story. Where they funded it.

Yes. A benchmark built, funded, or held by a party with a massive financial stake in the result is not automatically fraudulent, but it completely inverts the burden of proof. The incentives are structurally misaligned with the objective measurement.

And the fourth failure mode is Construct Mismatch. We're going to spend a lot of time on this later because it is so critical. But simply put, Construct Mismatch occurred when the benchmark simply does not measure your actual task.

Even a perfectly clean, independently held benchmark with zero contamination is completely useless to you if it measures the wrong thing. A high score on ResearchMath tells you absolutely nothing about whether a model will faithfully summarize a legal contract without inventing a clause. Okay, I have to stop us here and play the role of the skeptical buyer.

Go for it. If they aren't lying, but all these traps exist, contamination, flattering compute, conflicts of interest, construct mismatch, what are we supposed to do? It feels overwhelming. Tell our board we can't trust anything.

I mean, my team is already stretched thin. We certainly don't have the time or budget to rebuild an entire Frontier Math from scratch just to check a vendor's work. I can't assign six engineers for a month to validate one slide in a pitch deck.

Which is exactly the trap vendors rely on. That we're too busy. The assumption that verification requires mimicking their massive scale.

But you don't rebuild the whole benchmark. That is a fundamental misunderstanding of operational governance. You scale the evaluation down to a highly targeted day-sized task.

You reproduce a slice, not the whole benchmark, and you do it in one afternoon. In one afternoon. Yes.

This is the core operational framework of our deep dive today. The afternoon method. So how does a busy professional actually execute this afternoon method? Let's walk through the mechanics because this sounds like the exact toolkit our listeners need.

It starts with setting the target. You do not approach this by asking a vague question like, is this model good? Or is this model safe? Because... You can't measure that. Exactly.

Those aren't testable claims. You pick one specific vendor claim tied to a real procurement decision sitting on your desk. You quote the claim exactly.

You record what document it came from and you note the date. A moving target cannot be reproduced. Right.

If you don't anchor it, the vendor will just shift the goalposts later. Precisely. Once you have the target, you have to read the method.

You have to hunt down the footnotes. Yes. You read the primary source, the benchmark paper, the model card, or the technical appendix at the back of the white paper.

What are we looking for in there? You are looking for those four load-bearing parts we decompressed earlier. What was measured, out of what, at what setting, and by whom. What if they don't tell you? What if it's just a glossy marketing PDF and there is no technical appendix? If the methodology is undisclosed, if they won't tell you the setting or the denominator, that is a massive red flag.

Really? In fact, an undisclosed method is itself a critical governance finding. You write that down. An unreproducible claim is not evidence, it is just marketing.

Assuming they do provide the footnotes, we move to the center of gravity for the whole afternoon, building a small slice. You assemble just 20-50 items, but here is the key. You do not use their benchmarks task.

We use our own task. Exactly. You build these 20-50 items based on your own task.

If you are a logistics company, you use 30 real, messy routing requests from your dispatchers. Real data. Yes.

These items need to have known, verifiable, correct answers. And critically, you must write the pass-fail criteria before you run the model. Why before? Why not just see what it does first? Because language models are incredibly persuasive.

Oh, that's true. If you wait until after you see the model's outputs to decide what counts as a pass, the model will write something that is eloquently wrong, and it will subtly talk you into giving it a lenient score. I've definitely fallen for that.

We all have. You have to define what success looks like in cold, hard text before the model ever sees the prompt. Once you have your 30 items and your rigid criteria, you run it at two settings.

First, you run it at the vendor's claims setting, matching their compute and prompting as closely as you possibly can. And then you run it again at your actual deployment setting. You get two numbers.

And the delta between those two numbers is often the single most decision-relevant thing you will learn all year. We will talk about running a contamination probe on this slice in a moment, but the final step is to write the reproduction check. What does a reproduction check look like in practice? Is this a massive audit report? No.

It is a short, dated, one-to-two-page record. It contains the exact vendor claim, a description of your measured slice of 30 items, the gap between the headline number and your deployment reality, and a one-line verdict on whether the claim holds for your specific use case. Keep it simple.

Exactly. It is written in plain language so that a non-technical board member or a compliance auditor three years from now can read it and know exactly why you made the decision to buy or pass. Now I have to push back hard on the statistics here because I know what the data scientists listening are thinking.

I can guess what they're thinking. We are talking about running 30 items. A vendor like OpenAI or Anthropic or Google is running tens of thousands of items to get their 90% score.

Right. What can 30 items actually prove against 10,000? Aren't we just generating statistical noise? It is the most common objection and it is a completely valid statistical concern until you understand the specific intuitions behind why a targeted small slice run is so powerful. There are four distinct intuitions here.

Walk us through them. The first intuition requires you to actually agree with the data scientists. A small difference on a small slice is, in fact, just statistical noise.

So we admit it's noise. Yes. If the vendor claims 88% and your 30 item run gets 84%, that four point gap is well within the margin of error you would expect from chance alone on such a small sample.

You do not run into the boardroom and declare the vendor a liar over four points. You report it honestly as consistent with the headline within the limits of a small slice. You are not trying to prove a decimal point precision.

Wait. So if we are trying to prove the average score is wrong, where is the leverage in doing this? The leverage is in the second intuition. One catastrophic failure is a categorical finding that does not wash out with sample size.

Elaborate on that. What makes a finding categorical rather than statistical? Let's say you are buying an AI to draft responses for a healthcare portal. You run 30 real patient messages through it.

On 28 of those messages, it writes a polite, accurate response. But on two of those messages, it invents a highly confident, entirely fabricated medication instruction. It tells a patient to double their dosage of a heart medication when the doctor never said that.

It's a hallucination. A severe, life-threatening hallucination. Now, from a pure statistics perspective, a vendor might say, well, it could be 93% right, which aligns with our benchmark.

Yeah, 28 out of 30 is 93%. But as a buyer, you are no longer asking, is the average different? You are asking a categorical question. Does this model ever produce this specific, dangerous output on realistic input? And the answer is yes.

It happened twice. Wow. Yeah, that changes everything.

That is not a statistical anomaly to be averaged away across 10,000 easy questions. One clear instance of a safety-critical failure on real data is enough to halt a procurement. The harm happens to the specific person who receives that one output, regardless of how rare it is in the aggregate.

That makes total sense. You aren't checking their math. You are checking for fatal flaws.

What is the third intuition that makes 30 items work? Representativeness beats raw size. 50 items drawn to look exactly like your real work, including your messy, hard cases, ambiguous inputs, misspelled words, and diverse languages, tell you infinitely more about your deployment risk than 500 perfectly sanitized, easy items from a public dataset. So quality over quantity.

Exactly. If your real users span multiple dialects, and you omit those dialects to make the test fair or clean, you are just reproducing the vendor's blind spots. A small slice of reality is vastly superior to a massive slice of theory.

And the final intuition for the afternoon method? Human drift. What's that? Your own scoring can be the noisiest part of the exercise. A single person scoring 30 complex, qualitative items can drift, tire, or begin to read a get it over with.

Oh, I've been there. Yeah. You just want to finish the spreadsheet.

Right. To prevent your own drift, you must have a second person, a colleague, or subject expert independently score the ambiguous outputs. So synthesizing this, the afternoon method, isn't about estimating a perfect decimal percentage.

We are trying to see if the model fundamentally breaks when exposed to the harsh, messy reality of our specific business context. Exactly. It is a fast, honest, decision-relevant measurement, not a final academic audit.

Exactly. But to make that measurement honest, we have to ensure the model isn't just reciting a memorized script. Which brings us directly to the contamination probes.

Yes, this is crucial. If the model has simply regurgitated memorized answers from its vast training data, it is going to fail spectacularly when you give it novel, unseen data in production. We have to test on what the models cannot have seen.

How do we do that cheaply in one afternoon? There are three specific, cheap contamination probes you can run. The first and absolute strongest probe is using held out and private items. Meaning your own organization's unpublished data.

Real heavily redacted contracts. Real support tickets that live behind your firewall. Real internal invoices.

Because this data has never, ever been published on the open web, nothing the model saw on its training scrapers could possibly include it. Therefore, if the model scores highly on summarizing these private documents, it is proof of actual reasoning capability, not just memory. Exactly.

But sometimes you need to test general knowledge, not just internal documents. That is where the second probe comes in. Post cutoff items.

How does the cutoff date work in this context? Every model has a stated training cutoff date in its technical documentation. For instance, it might say training data up to December 2024. You build your test items using news events, new legal regulations, or financial documents created in January 2025 after that cutoff date.

But I've heard cutoff dates are somewhat fuzzy. They are. It's a strong probe, but not absolute.

Because cutoff dates are approximate, and sometimes a vendor will do lightweight fine-tuning later that folds in newer material without updating the official date. However, if a model aces a public benchmark about, say, historical regulatory law, but suddenly stumbles and hallucinates when asked about an equivalent regulation passed just last month, it strongly signals that its high score was leaning heavily on memorization. The third probe is my absolute favorite, because it is just so incredibly clever.

The perturbation probe. Oh, yeah. That one's brilliant.

The perturbation probe is where you take a public, known, benchmark-style item, and you systematically change its surface details without changing its underlying structural difficulty. You swap the numbers, you rename the people and places, you rephrase the question from active to passive voice. And if the model is actually reasoning, it shouldn't care if the apples are now oranges or if the five is now a seven.

Exactly. A model that is genuinely reasoning through the logic of the problem will score about the same on the perturbed version. But a model that simply memorized the topography of the original answer key will often drop sharply.

And we have hard data on this. Yeah, the source material has a brilliant, illustrative example of this. In 2024, a research team at Apple conducted a study they called GSM Symbolic.

Yes, GSM Symbolic. They took standard grade school math problems that are universally used to benchmark AI models. The original benchmark is called GSM-AK.

The researchers essentially just altered the surface details. Instead of, John has five apples, gives two to Mary, how many does he have, and whatever, they wrote, Liam has seven oranges, gives three to Sophia. The mathematical difficulty remained identical.

Basic subtraction. And the results were incredibly revealing. The accuracy of the leading frontier models dropped significantly.

The Apple researchers proved that the apparent reasoning on the original benchmark was, to a large extent, just pattern matching to memorized forms. I want to dig into that pattern matching because this is where the predictive text analogy is so powerful. It really is.

The LLM's neural network isn't actually learning the concept of subtraction, is it? It is acting like a massively scaled-up version of the predictive text on your smartphone keyboard. That's a great way to put it. It hasn't learned math.

It has mapped the topography of how numbers usually sit next to each other on the internet. That is precisely what is happening. If it sees the phrase, John has five apples, its internal weights fire based on billions of internet pages where that exact phrasing appears, and it statistically predicts the next tokens, which happen to lead to the correct answer.

But when you change the names and numbers, those specific statistical pathways aren't as strong, and the illusion of reasoning shatters. The model is forced to rely on generalized logic, which it often lacks. It's the ultimate open book test trick.

If a student genuinely understands the mathematical formula, changing the numbers on the test won't stop them at all. But if they just memorize these specific answers from a stolen copy of last year's test, they will fail spectacularly when you swap a five for a seven. That's what the perturbation probe does.

It forces the model to show its actual work. So let's assume you have secured, clean, private items, and you are confident the model hasn't memorized the test. Okay, assuming that's done.

You still have a massive hurdle to clear before you can trust a number. The model might be running on a highly specialized steroid setting that your organization simply cannot afford to deploy. You must run your slice at the setting you will actually ship.

Let's decode the footnotes where this inflation hides. We touched on compute budgets earlier, but let's look at the mechanics of these settings. First up, as a classic statistical trick, pass at 1 versus pass at k, which is also commonly known as best of n. Pass at 1 means the model is given exactly one attempt to solve the problem, and its single answer is scored.

Best of n means the model tries repeatedly. Over and over. Yes.

Maybe it generates 16, or 64, or even 1,000 different answers to the same question. Then an external evaluator, usually a separate code script, picks the luckiest, best run out of that batch to count as the official score. Let's explain the math behind that, because it's staggering.

It really is. If a model only has a 10% chance of getting a complex coding problem right on the first try, its pass at 1 score is 10%. But if you roll a 10-sided die 64 times, the statistical probability that you land on the right answer at least once approaches near certainty.

Exactly. A best of 64 score can look astronomically high. It might jump from 10% to 85%.

But there's a catch. A huge catch. If your actual deployment is a customer service chatbot or a real-time copilot for a software engineer, it only allows the model one single attempt to answer a user.

The user isn't going to wait for 64 different answers to be generated and scored. Right. Therefore, that best of N score is completely and utterly irrelevant to your deployment reality.

Next is the compute budget. We mentioned the O3 preview, but how far can compute inflation go? Very far. The sources cite the ARCAGI challenge, also known as the ARC Prize, which ran extensively through 2024 and 2025.

Okay, tell us about that. This is a fascinating benchmark of abstract visual reasoning puzzles. When a Frontier model reached a remarkably high score on it, the headline was everywhere.

I remember seeing that. But the widely shared caveat among researchers was that this result required massive, almost unsustainable compute costs per task to generate the solution. Because they were using test-time compute, letting the model think and simulate thousands of pathways before committing to an answer.

Right. And when that same model was run on a lower, fixed, affordable compute budget, the kind a normal company would use, the score was substantially lower. You always have to ask, can I afford the per-query server costs that generated this headline number at my production volume? Then we have tools and scaffolding.

We talked about agentic loops with SWE Bench, but this also includes hidden prompts, right? Yes. A benchmark run might include heavily engineered hidden system prompts. Like what? An undisclosed instruction as simple as, think step-by-step before answering, hidden in a prompt, can lift a model's score on a logic test by 10 to 20 points.

If the vendor used that prompt, but you don't use it in your application, you will never see that performance. And finally, there is Retrieval Augmented Generation, or RAG. Let's explain how RAG actually works mechanically, because it is the biggest buzzword in enterprise AI right now.

Mechanics-wise, RA is when the model is connected to a database, usually a vector database. Before the model answers a user's prompt, the system performs a semantic search of the database, retrieves relevant documents, and pastes those documents directly into the model's prompt behind the scenes. So the model gets to read the documents before it answers.

It's an open book test. Exactly. Yeah.

And here is the profound warning for buyers, RAG can artificially mask contamination and inflate capability scores. How so? If the benchmark setup lets the model read the correct answer from a beautifully curated, perfectly formatted document that was designed specifically for the test, the model will look incredibly capable. Ah, I see.

But in the real world, my corporate database isn't perfectly formatted. It's 10 years of messy PDFs, conflicting policies, and undocumented jargon. Precisely.

If the benchmark gives the model a clean document that you will never possess in your messy production environment, the score is an illusion. The model looks like it possesses deep reasoning, but it's just reading the answer off a sanitized cheat sheet. Okay, here is the ultimate reality check.

Reproduction by demand. What happens if we literally cannot run the afternoon method because we cannot access the model? What do you mean? What if the model is behind a strict sales wall? The API isn't public yet, and the vendor says, we can't let you run your own data. Just look at our curated demos in our secure sandbox environment.

If you connect this to the bigger picture of corporate governance and fiduciary duty, you do not stop at that wall. If you cannot run it yourself, you make the vendor run your 30 items at your deployment setting right in front of you. Reproduction by demand sounds great in theory, but in practice, aren't these vendors going to push back? They have the massive models, they have the compute, they have the leverage here.

Why would they agree to run my 30 messy questions? Because you have the budget. You are the buyer. You hand them your 30 items and your pre-written grading criteria.

You explicitly require them to use the shipped configuration, not a high compute preview and not best of 64. And critically, you require the raw textual outputs, not just the summary average score. You need to see the actual text it generates so you can spot the hallucinations.

A vendor confident in their product will usually accept this as an easy way to win a lucrative enterprise deal. And what if they say no? What if they refuse to run our items? If they refuse, you treat that refusal as a critical terminal governance finding. A deal breaker.

You write it into your reproduction check. An unfalsifiable claim is a weak claim. A vendor who refuses to run your items on your task at your setting has told you everything you need to know about how the model will actually perform.

You do not sign the contract, you walk away. That shifts the power dynamic entirely. It takes the illusion of benchmark authority and grounds it in commercial leverage.

Now let's move to the final and perhaps most profound conceptual shift of this deep dive. Construct validity. Yes, this is the big one.

Because we've secured clean items, we've forced honest settings, we've probed for contamination. But what if the benchmark itself measures the wrong thing entirely? This is the most sophisticated and pervasive failure mode in AI procurement. Construct validity asks a very simple, devastating question.

Does the thing being measured actually resemble the thing you care about? A benchmark predicts your task only to the exact degree it resembles your task. Let's provide some vivid examples from the sources, because the AI industry is flooded with impressive-sounding benchmarks. We have GPQA, which stands for graduate-level Google-proof Q&A, very hard science question.

Very impressive sounding. We have ARCAGI, which is abstract visual reasoning. We have MedQA for medical board exams, and LegalBench for legal reasoning.

Every single one of those is a real, rigorous, respectable benchmark. But a high score on graduate-level quantum physics questions in GPQA does not predict whether a model will faithfully summarize a messy emotional customer complaint without inventing a fake company refund policy. A high score on LegalBench does not mean the model can handle your specific idiosyncratic internal compliance workflows.

The construct of answering a multiple-choice legal question is fundamentally different from the construct of drafting a novel contract clause based on messy client notes. And what about leaderboards? Everyone loves a leaderboard, they feel so democratic. They do, but they have their own construct issues.

The sources highlight Chatbot Arena, run by an organization called LMSYS. For a long time, it relied entirely on human preference votes. Two models generate an answer, and a human clicks which one they like better.

Sounds like a great way to measure quality. Until you understand human psychology, human preference is heavily confounded by style. In 2024, LMSYS had to introduce style control to their leaderboard.

Why? What happened? Because data showed that longer, nicely formatted, confident-sounding answers with bullet points tend to win human votes, even if the actual information in those bullet points is factually wrong or dangerous. Humans are easily tricked by a confident tone. So a model that wins a preference leaderboard based on style could still fabricate catastrophic errors on your specific narrow task.

You also have to address language and cultural context, because this is a massive blind spot. It is critical. Most widely cited benchmarks are built in English, and they are heavily weighted toward the cultural, historical, and legal contexts of the United States.

If your organization operates in Bahasa Indonesia or Swahili, or operates under the regulatory regime of the European Union or India, a U.S.-weighted English benchmark guarantees construct mismatch. The model might be a genius in English and completely hallucinate in Swahili. Reproduction on your own task in your own language is not optional due diligence.

It is the absolute minimum requirement to know anything at all. To bring all of these concepts together, contamination settings, construct validity, the afternoon method, I want to walk through an extended immersive scenario from the sources. This is the story of Iris.

This is a great case study. She is a healthcare AI governance lead for a large regional health service. Let's set the stage.

It is a Tuesday, and she has a major procurement decision due on Friday. A vendor is aggressively pitching a new AI model to draft first-pass replies to patient portal messages. These are the messages patients send their doctors at 2 a.m. Right.

Very sensitive data. The vendor's pitch deck features a massive, bold slide boasting an 88% score on a public medical QA benchmark. Her director, who is feeling the pressure to modernize, catches her in the hallway and says, Iris, just confirm that 88% is legitimate so we can sign the contract and get moving.

This is the exact moment where governance usually fails. Iris recognizes immediately that being asked to confirm a number is a trap. Yes, confirmation bias.

If you set out to confirm a number, cognitive bias ensures you will find confirmation. She reframes the request entirely. She tells her director she will not confirm it.

Instead, she will reproduce a slice of it on their own messages at their deployment setting by Thursday. So she starts step two of the afternoon method. She reads the footnotes.

She looks at the primary source for the 88% benchmark. And she immediately spots a massive construct mismatch. So what does she find? The benchmark the vendor cited is entirely composed of multiple-choice medical board exam questions.

Things like, what is the mechanism of action for drug X, A, B, C, or D? But her actual task, the thing the hospital is buying the AI to do, is writing free text replies to messy, worried, ambiguous patient messages. Those are totally different things. Right.

Picking the letter C on a cleanly formatted exam is a completely different construct than writing a safe, empathetic reply to a patient who types, my chest feels tight after the new tablet. Should I keep taking them? Not only is there a construct mismatch, but the benchmark is old and publicly available on the internet, so contamination is highly likely. The model might have just seen those exact multiple-choice board questions on a medical student forum during its training.

So Iris executes the afternoon method. She doesn't use the vendor's multiple-choice questions. She builds a 30-item slice using real, heavily redacted patient messages from her clinic's own secure archives.

That's the private data probe. Exactly. These are private items, so they act as a contamination probe.

Then she sits down with the senior triage nurse. Together, before they run the model, they write down exactly what a safe reply must and must not contain for each of the 30 messages. She establishes the criteria before the model can persuade her.

Then she runs it at two settings. Yes. She runs it once at the vendor's flattering setting, allowing it best of many attempts to see if it can hit the 88%.

And then she runs it at the deployment setting. Her clinic can actually afford one single attempt with a plain system prompt, because that is how the portal will actually work. The results are where the theory becomes practice.

At the deployment setting, the average score drops meaningfully. But critically, and this is where Intuition 2 comes in, the power of a categorical finding on two out of the 30 messages, the model does something terrifying. What does it do? It invents a highly confident, completely fabricated medication instruction.

It tells one patient to stop taking a critical medication based on a misunderstood symptom. Two fabrications out of 30. Now a vendor rep might look at that and say, well, 30 is a tiny statistical sample compared to our tens of thousands of tests.

You can't judge the model on two edge cases. But Iris understands the statistics of a small slice. She isn't estimating an error rate.

She is asking a categorical question. Does this model ever invent a clinical instruction on a real messy patient message under our deployment conditions? And the answer is unequivocally yes. It happened twice in a single afternoon.

If they deploy this to 10,000 patients a day, that failure mode scales immediately. So Iris writes her one-page reproduction check. On Friday morning, the director puts the vendor's glossy 88% slide up on the boardroom screen.

And Iris puts her one-page document on the table next to it. She says the 88% is real. But it is on a different task, at a different compute setting, on multiple-choice questions it has likely already seen.

On our actual messages at our affordable setting, it invented dangerous medication instructions twice in 30 attempts. Iris didn't waste breath debunking the vendor's 88%. She simply rendered it irrelevant.

She replaced an argument over a marketing slide with a measured fact grounded entirely in their deployment reality. And the outcome? The contract is halted. They demand the vendor pass their specific fabrication test on their own data before any pilot begins.

That is the power of the afternoon method. It takes you from being a passive consumer of vendor claims to an active, empirical evaluator of your own reality. To synthesize everything we've covered in this deep dive, I want to leave you with the expert mental model.

When you see a benchmark score on a slide deck, run these five questions before you believe it. 1. Out of what? Which items? Which version of the weights? Public or holdout? 2. Under which setting? One attempt or best of many? High compute preview or shipped configuration? 3. Could the model have seen it? Is it an old public benchmark? Or have you run a perturbation or private data contamination probe? 4. Who built and funded it? If it's an interested party creating their own homework, weight it lower and demand independent reproduction? And 5. The most important. Does it measure my task? Does it have construct validity for the actual messy job I need this AI to do? If you ask those five questions, you decompress the claim, you strip away the illusion of precision and you see the structural, highly subjected choices underneath.

We have a specific Monday morning action for you, the listener. This coming Monday, I want you to look at the single largest AI procurement decision currently sitting on your desk. Take the vendor's headline benchmark claim.

Refuse to take it on faith. Block out one afternoon. Spend that time running 20 to 50 of your own highly representative items through the model at the exact setting you will actually deploy.

Write down your pass fail criteria beforehand, run a quick contamination probe and write down exactly what happens in a one page reproduction check. This is how you build an organizational defense against information asymmetry. The power dynamic in AI procurement usually runs entirely toward the vendor.

They have the model, they have the compute, and they have the massive marketing budget. And you just have a slide. Right.

This method restores the symmetry required to make an honest fiduciary decision. It really does. We want to leave you with one final provocative thought from the source material.

Think of a vendor's benchmark slide as a photograph they took in the flattering lighting they chose of a pose they liked. I love this metaphor. Your reproduction check is a photograph you take of your own work in the harsh, unforgiving light that you will actually operate in.

When the board asks you how you knew the AI was safe to deploy, which photograph do you want to hand them?