Reproducing a claim: testing a vendor benchmark yourself in an afternoon
The short answer
A benchmark score is a claim, not a fact
It is the output of a procedure with choices in it (which task, which items, which setting, scored by whom), and changing the choices changes the number with no change to the model. The same o3 scored about 25 and about 10 percent on FrontierMath under different settings and reporters (Epoch AI, 2024 to 2025). Decompress every score into its four parts before it moves you.
What you will be able to do
- Analyze a vendor benchmark claim by decomposing it into its four load-bearing parts (what was measured, out of what, under which setting, and by whom) so that a single headline percentage becomes a set of checkable questions.
- Identify the four ways a benchmark number misleads without anyone lying: contamination (the model saw the test), a flattering configuration (a setting you will never deploy), a conflict of interest in who built or funded the benchmark, and construct mismatch (the benchmark does not measure your task).
- Design an afternoon reproduction: a small, honest slice of the vendor's evaluation run on your own representative task, at the vendor's claimed setting and at the setting you would actually ship, so the gap between the two becomes visible.
- Run a contamination probe (held-out items, post-cutoff items, and perturbed variants) that reveals whether a headline score rests on genuine capability or on the model having seen the questions before.
- Distinguish a benchmark's reported score from its relevance to your organization, using construct validity: whether a number on research-level math, or coding puzzles, or trivia, predicts anything about the task you would actually buy the model to do.
- Produce a reproduction check on one vendor claim that matters to a live decision in your organization, an artifact that feeds your weekly frontier watch in (see Topic 12.3), your successor's briefing in (see Topic 12.5), and the dossier in (see Topic 13.1), and that strengthens the buy-or-build case in (see Topic 8.3).
- Apply the honesty rule of reproduction: report the number you actually got, including when it matches the vendor and including when your own method is too weak to conclude anything, because a reproduction that quietly rounds toward the answer you wanted is worse than no reproduction at all.
The ideas, one by one
You reproduce a slice, not the whole benchmark
The point is not to recreate the vendor's exact figure but to convert a claim you were asked to accept into a number you watched appear, on 20 to 50 of your own items, in one afternoon. A slice you measured beats a headline you accepted, and it fits in a day.
Four failure modes inflate a number without anyone lying
Contamination (the model saw the test), a flattering configuration (a setting you will never deploy), conflict and provenance (an interested party graded it), and construct mismatch (it does not measure your task). A strong reproduction probes all four, and each has a concrete home in the FrontierMath case.
Test on what the model cannot have seen
Re-running public items reproduces possible contamination, not capability. Held-out private data, post-cutoff material, and perturbed variants are the three cheap probes that reveal whether a high score is genuine skill or memorized answers.
Run at the setting you will actually ship
Most inflation hides in the footnotes: best-of-N versus one attempt, high-compute preview versus shipped, tool use and prompt scaffolding you do not have. Reproduce the claim at both the vendor's setting and your deployment setting, and treat the gap as the decision-relevant fact.
Relevance is decided by construct validity, not by rank
Even a clean, honest, independently held benchmark is useless if its task does not resemble yours. A model that tops a math or trivia leaderboard can fabricate on your contracts or your patients. The only number with guaranteed relevance to your decision is the one you measure on your own task.
Spend the afternoon where the decision is big, not where the claim is loud
Reproduction is a cost, so route it by stakes: a claim that changes a large purchase, a patient-facing or citizen-facing deployment, or a switch from a working system earns the full afternoon, while a claim that feeds only a newsletter earns just the five quick questions. The skill is knowing which claims clear the bar, and it is the habit your weekly frontier hour builds (see Topic 12.3).
The reproduction check is evidence, and it persists
The dated one-page record of what the vendor claimed and what you measured feeds your weekly frontier watch (see Topic 12.3), your successor's briefing (see Topic 12.5), the dossier (see Topic 13.1), and the vendor-selection case (see Topic 8.3). Report the number you got, including when it matches and including when your slice is too small to conclude, because a reproduction that rounds toward the answer you wanted is worse than none.
When you cannot run the model, make the vendor reproduce it in front of you
Hand over your slice, your pre-written criteria, and your deployment setting, and require the raw outputs so an average cannot hide a fabrication. A vendor confident in a claim welcomes this; a refusal is a finding you write down. Reproduction by demand is how a buyer restores enough symmetry to decide honestly against a party that holds the model, the benchmark, and the marketing budget.
A small slice concludes some things and not others, and honesty is knowing which
One clear instance of a dangerous output is a categorical finding that no sample size washes out; a few-point difference in an average is noise on a few dozen items. Choose representative items over merely more items, state your sample size and uncertainty next to your numbers, and when the stakes need more, name the larger slice rather than overclaiming from the small one.
You already own the skill; this topic just aims it outward
Reproducing a vendor benchmark is the same evaluation discipline you built in Module 4 (see Topic 4.2) and the same distrust of a single number you earned by breaking your own model in Module 1 (see Topic 1.6), now pointed at a claim someone else wants you to believe. The new caution is only this: a model you did not build and cannot see inside adds contamination, a flattering configuration, and provenance to the list of things that can inflate its number.
Listen to it
The same lesson, talked through by two hosts as episode 91 of the podcast.
Real cases
These examples show benchmark claims meeting reproduction, with the reasoning stated plainly. The anchor case comes first.
Example 1: FrontierMath and the 25-to-10 gap (Epoch AI and OpenAI, 2024 to 2025). OpenAI announced o3 with a headline of roughly 25 percent on Epoch AI's FrontierMath benchmark, a set of original, unpublished research mathematics problems (announcement 20 December 2024). It later emerged that OpenAI had funded the benchmark and had access to most of its problems and solutions, disclosed the same day in a paper update, with several contributing mathematicians saying they had not been told (TechCrunch, 19 January 2025). When Epoch AI ran its own independent evaluation, the shipped o3 scored around 10 percent (Epoch AI, April 2025). Reproduction lesson: the headline was a high-compute preview reported by the funder of a benchmark the funder could see into; the independent number at the shipped setting was less than half. Every failure mode from Section 3D appears here at once, and the corrective was independent measurement.
Example 2: MMLU and benchmark aging (Hendrycks and colleagues, 2021, ongoing). MMLU (Massive Multitask Language Understanding) is a widely used multiple-choice benchmark across many subjects. Because it has circulated on the open web for years, it is a standing example of contamination risk: fragments of it appear in training corpora, and scores tend to creep upward for reasons unrelated to reasoning. Researchers responded by building refreshed and harder variants and cleaned versions to reduce leakage. Reproduction lesson: an old, popular, public benchmark is the most contaminated kind; a high score on it should be checked against held-out or post-cutoff items before it is believed. Established.
Example 3: SWE-bench and configuration effects (Jimenez and colleagues, 2023 to 2024). SWE-bench measures whether a model can resolve real software issues drawn from open-source repositories. Reported scores on it vary widely depending on the surrounding setup: the agent loop, the retrieval of relevant files, the number of attempts, and the exact subset used (a smaller "verified" subset scores differently from the full set). Reproduction lesson: for a benchmark like this, the number is almost meaningless without the configuration, and two vendors quoting SWE-bench scores may be describing entirely different setups. Ask which subset and which scaffolding before comparing. Established that configuration matters; treat any specific leaderboard figure as fast-moving.
Example 4: The perturbation probe on grade-school math (Apple research team, GSM-Symbolic, 2024). Researchers generated altered versions of standard grade-school math problems, changing names and numbers while keeping the difficulty, and reported that several leading models' accuracy dropped on the altered versions. Reproduction lesson: perturbation is a cheap, powerful contamination probe you can run yourself in an afternoon, and a score that falls when the surface changes is a score that was partly memorization. Treat the specific magnitudes as emerging; treat the technique as sound and reusable.
Example 5: Leaderboard versus deployment on the same model name (LMArena, 2025). A public arena that ranks models by human preference votes was found to have ranked a specially tuned build of a model far higher than the version users could actually download, the same base model with different tuning producing different results (see Topic 1.4) for the full treatment. Reproduction lesson referenced here only: "which exact build" is part of "out of what," and a leaderboard rank on a tuned submission does not transfer to the shipped weights. This event is owned by Topic 1.4; here it stands as a reminder that the denominator includes the model version, not just the test items.
Example 6: A clean, honest benchmark that still does not fit (construct mismatch, generic). Consider a public-sector team evaluating a model to answer citizens' housing-benefit questions. A vendor presents a strong score on a graduate-level science question benchmark (GPQA-style). The benchmark is real, independently built, and uncontaminated. It is also completely irrelevant: nothing about answering hard physics questions predicts whether the model will fabricate a benefits eligibility rule under a citizen's pressured, ambiguous phrasing. Reproduction lesson: the team's afternoon is better spent building a 30-item slice of real (redacted) benefits questions with known correct answers than arguing about the vendor's benchmark at all. The construct is the whole game.
Example 7: The abstraction-and-reasoning challenge and the compute footnote (ARC-AGI, ARC Prize, 2024 to 2025). ARC-AGI is a benchmark of abstract visual-reasoning puzzles associated with the researcher Francois Chollet and run as a public prize, designed so that memorization does not help and each task requires reasoning from a few examples. When a frontier model reportedly reached a high score on it in late 2024, the widely shared caveat was that the strongest result came at a very high compute cost per task, again the flattering-configuration pattern, and that a lower-compute run scored substantially less. Reproduction lesson: even a benchmark built specifically to resist contamination still carries a setting, and the headline number and the affordable number can be far apart. Treat the specific scores as fast-moving; treat the compute-versus-score tradeoff as the durable lesson.
Example 8: Preference leaderboards and the confound of style (human-preference arenas, generic). Public arenas that rank models by which answer humans prefer in blind comparisons are useful, but researchers have noted that human preference is confounded by style: longer, more formatted, more confident answers tend to win votes independent of correctness. The operators of Chatbot Arena, one of the best known such arenas (LMSYS), introduced "style control" analysis in 2024 specifically to separate substance from formatting effects in the votes, which is itself an acknowledgment that raw preference rank was being distorted. Style is not the only confound: which prompts were sampled, which models were entered, and who the voting population is can all shift a rank without any change in which model is more correct or safe. Reproduction lesson: a "most preferred" ranking measures what a particular population liked on a particular set of prompts in a blind vote, which is not the same as what is correct, safe, or right for your task, and it is unlikely to predict performance on a narrow, high-stakes task the arena's prompts never covered; a model that wins on style could still fabricate on your work. When the metric is preference, ask what the preference is actually tracking, on what prompts, and by whom, before you let it stand in for quality.
Where people go wrong
- "A benchmark score is an objective property of the model." No. It is the outcome of a procedure with choices in it (which items, which version, which setting, scored by whom). Change the choices and the number changes with no change to the model, as the same o3 scoring 25 and 10 on FrontierMath shows. Treat every score as a claim to be decompressed, never a fact to be accepted.
- "Reproducing a benchmark means recreating the vendor's full number." It does not, and believing it does is why people never reproduce anything: the full job looks impossible, so they skip it and take the headline. Reproduction means measuring a representative slice on your own task at your own setting. A slice you measured beats a headline you accepted, and a slice fits in an afternoon.
- "If the vendor built the benchmark, the score is worthless." Too strong. A self-built or self-funded benchmark is not disqualified; it is a claim that inverts the burden of proof and demands independent reproduction first. FrontierMath was serious work with a genuine holdout design; the failure was in presentation and access, not in the benchmark existing. Weight self-reported scores lower and reproduce them, rather than dismissing them unexamined.
- "A higher leaderboard rank means a better model for us." Only if the leaderboard's task resembles your task. A model that tops a math or trivia leaderboard may fabricate on your contracts or your patient messages. Construct validity, not rank, decides relevance, and the only number with guaranteed construct validity for your decision is the one you measure on your own task.
- "We ran the vendor's public benchmark items ourselves, so we reproduced it." Re-running public items mostly confirms the model can do items it may have memorized. That is not reproduction of capability; it is reproduction of possible contamination. Real reproduction tests on clean items the model could not have seen: your held-out data, post-cutoff material, or perturbed variants.
- "The setting details in the footnote are technical trivia." The footnote is usually where the inflation lives: best-of-N versus one attempt, high-compute preview versus shipped, tool use and prompt scaffolding you do not have. The o3 headline was a preview configuration. Read the setting before you read the number, because the setting is what makes the number comparable to your world or not.
- "Our reproduction should confirm the vendor if the vendor is honest." Not necessarily, and expecting confirmation is a bias that corrupts the measurement. Even an honest vendor reports a flattering setting on a possibly contaminated, possibly mismatched benchmark. Your job is to report the number you actually got, including when it undershoots the headline for legitimate reasons, and to state honestly when your own slice is too small to conclude.
- "If reproduction is inconvenient, the five quick questions are enough." The five questions (out of what, which setting, could it be contaminated, who built it, does it measure my task) are enough for low-stakes claims and are always the first move. But when a decision is large and the risk is real, "we asked good questions" is not "we measured it." The whole point of the afternoon is that a real decision deserves a number you watched appear, not a claim you interrogated and then believed anyway.
- "A number I measured on 30 items is as reliable as the vendor's number on thousands." No, and pretending so is its own failure. A small slice has real statistical limits and can miss failure modes that appear only at scale. The discipline is to report your slice honestly, including its size and its uncertainty, and to treat a dramatic result (like a fabricated clinical instruction) as decision-relevant while treating a two-point percentage difference as within the noise of a small sample.
- "The vendor would not run our items, but their benchmark score is strong, so we should proceed." A refusal to reproduce a claim on your task at your setting is itself a finding, and a serious one. An unfalsifiable claim is a weak claim: a vendor confident in a number usually welcomes an easy way to prove it, and one who will only show curated demos has told you not to trust the headline. Record the refusal in the reproduction check as plainly as a measured result.
- "We should build our slice from easy, clean cases so the test is fair." A slice that omits your hard and ambiguous cases flatters the model exactly the way a curated benchmark does, and reproduces the vendor's mistake instead of catching it. The leverage of an afternoon comes from including the cases where a wrong answer costs the most (ambiguous inputs, cases that must escalate, tempting-but-wrong answers), because those are where deployment risk actually lives.
- "Reproduction is about catching dishonest vendors." Mostly it is not. The FrontierMath number required no lie: a flattering setting, an interested reporter, and a possibly contaminated benchmark produced a misleading headline with everyone acting within the letter of honesty. Reproduction protects you against the ordinary structure of the AI market, not just against bad actors, which is why you run it even on vendors you trust.
Questions people ask
- What is benchmark?
- A fixed set of tasks with known answers used to measure and compare AI models. A benchmark score is the outcome of running a model against those tasks under a specific procedure, not an intrinsic property of the model. More on Benchmark
- What is reproduction check?
- The artifact this topic produces. A dated, one-page-to-two-page record stating a vendor's exact benchmark claim, the slice you reproduced on your own task, the number you measured at the vendor's setting and at your deployment setting, the contamination result, and a one-line verdict for your use.
- What is frontierMath?
- A benchmark of original, unpublished research-level mathematics problems built by Epoch AI, on which OpenAI's o3 was announced (20 December 2024) to score about 25 percent; independent evaluation later placed the shipped model around 10 percent (Epoch AI, 2025). The benchmark was funded by OpenAI, which had access to most problems, disclosed late. The anchor case of this topic.
- What is contamination?
- When a benchmark's questions (or close paraphrases) appear in a model's training data, letting the model score high by memory rather than capability. The most common cause of inflated benchmark numbers, and the reason benchmarks like FrontierMath use unpublished problems.
- What is holdout set?
- A portion of a benchmark deliberately kept private and unseen by the model builder, reserved for independent scoring. The presence and use of a holdout is what makes a self-funded benchmark's score believable.
Keep going
This lesson builds AI vendor due diligence and third-party risk, and that page shows the roles that hire for it. Every Certified AI Governance Professional (CAIGP) lesson.
Watch the lesson and prove it
Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.
Watch the lesson and prove it →