The honest ROI: measuring what the AI actually changed at your organization, not what the vendor promised
The short answer
The vendor's number is a sales artifact
Any ROI figure you did not compute yourself was built to win a decision, not to be true. The honest number can only equal or fall below it once tested, never rise, because every untested part was chosen to flatter.
What you will be able to do
- Decompose any AI ROI claim into its four load-bearing parts (the benefit claimed, the baseline it is measured against, the attribution to the AI, and the costs netted out) and locate which part is doing the misleading.
- Distinguish a benefit the AI actually caused from a change that would have happened anyway, using an explicit baseline and counterfactual reasoning.
- Separate a vanity metric (response time, tickets closed, adoption) from a value metric (money saved, revenue gained, risk reduced) and explain why the first is often reported in place of the second.
- Assemble the full cost of ownership of an AI system, including the costs vendors and champions routinely leave out (human supervision, integration, licensing, monitoring, drift, incident risk, severance, and rework).
- Detect metric gaming, especially the deflection-versus-resolution trap, where a system is credited for closing work it did not actually complete.
- Adjust a headline ROI for risk and uncertainty, producing a value range with a stated confidence rather than a single hero number.
- Set the boundary and the time frame of a measurement explicitly, distinguishing a cost that was removed from a cost that was merely shifted onto another team, your customers, or the public.
- Produce a one-page honest ROI measurement for one AI system in your own organization that a chief financial officer (CFO), the senior executive responsible for the money, could not dismantle in the first five minutes.
The ideas, one by one
Every ROI claim has four joints
Benefit, baseline, attribution, and costs. Analyze means naming which joint fails, by how much, and in which direction the true number moves once it is fixed. Distrust is not analysis; structured decomposition is.
Measure against the counterfactual, not the past
Honest ROI is the world with the AI minus the world you would have had without it over the same period. Crediting the AI with change that would have happened anyway is the most common inflation.
Activity is not value
Tickets closed, documents generated, code accepted, and leads scored are activity. They count only if the outcome behind them actually occurred. A system optimized for activity can look excellent while destroying value.
The deflection trap is expensive
A contact "resolved" may mean the problem was solved, or the customer gave up, or the customer churned. Only the first is value; some of the rest are large negatives counted as positives. Always pull the outcome behind the activity.
Price the whole iceberg
Total cost of ownership is dominated by costs below the sticker price: supervision, integration, monitoring, inference that scales with use, incident risk, and transition. A benefit is only real net of all of them.
Report a range, not a hero number
When attribution and counterfactual are estimated, the honest output is a low-to-high band with stated assumptions. A defended range earns more trust than a bare point.
Risk-adjust the return
Two systems with the same expected value are not equally valuable if one carries a tail risk of a fine, an outage, or a reputational event. Name what could turn the return negative and watch its leading indicators.
A negative honest ROI is a success
The measurement that justifies killing a value-destroying system is worth more than an optimistic number that keeps it alive. Honesty runs in both directions.
When you measure decides what you see
State the window, the payback status, and the drift assumption behind any figure. A short window flatters, a pre-payback reading understates, and a launch-day accuracy assumed to hold forever overstates. A number with no stated time frame is not yet a measurement.
Ask whether cost fell or only moved
An AI can improve your ledger by exporting cost to another team, your customers, or the public. Draw the ROI boundary wide enough to catch the cost you may be shifting, and call a transfer a transfer, not a saving.
This worksheet is load-bearing
The honest ROI measurement you build here feeds the investment memo, the portfolio review, the hostile-board defense, and the capstone audit. Soft here means soft everywhere downstream.
Listen to it
The same lesson, talked through by two hosts as episode 60 of the podcast.
Real cases
These examples show honest-ROI reasoning applied to real, sourced situations. The reasoning is the transferable skill; the numbers belong to the organizations that lived them.
Example 1: Dukaan, the headline that invites its own audit (India, 2023). Suumit Shah, chief executive of Dukaan, reported replacing about 90 percent of support staff with an AI chatbot and cutting support cost by about 85 percent, alongside near-instant first response and resolution falling from 2 hours 13 minutes to 3 minutes 12 seconds (CNN Business, 12 July 2023). Apply the four parts. Benefit: an 85 percent cost cut is a value metric, good, but the resolution-time figure is an activity metric vulnerable to the deflection trap. Baseline: 85 percent against which support cost, and would any of it have fallen without the AI? Attribution: the cut coincides with a layoff, so part of the saving is simply fewer salaries, a decision that could have been made with or without a bot. Costs: the figure appears gross, with no visible netting of supervision for the harder tickets, severance, or the risk of a customer-facing failure. None of this proves the AI was a bad decision for Dukaan; the point is that the public number, as stated, cannot tell you whether it was, and a year later the company reaffirmed the efficiency claims without publishing the netted-out ledger that would settle it (follow-up coverage, 2024). The headline is an invitation to do the honest measurement, not a substitute for it.
Example 2: The MIT NANDA finding, most pilots show no measured P&L impact (United States study, 2025). "The GenAI Divide: State of AI in Business 2025" (MIT NANDA, August 2025) found that roughly 95 percent of enterprise generative-AI pilots produced no measurable profit-and-loss impact, despite very large aggregate investment. The honest-ROI reading is not "AI does not work." It is that the gap between adoption and measured value is enormous, and that organizations reporting AI success are frequently reporting activity and adoption, not P&L. Treat this as an established directional finding (most pilots show no measured return) with the specific percentage attributed to the study, not asserted as universal. It is the strongest available evidence that honest measurement, not more pilots, is the scarce discipline.
Example 3: The offshore-labor reclassification that flatters automation ROI (pattern, globally documented). A recurring pattern across AI deployments is crediting a system with savings that actually came from moving work to cheaper humans, sometimes the very humans the "AI" quietly relies on. The deep enforcement cases of this pattern (a checkout "AI" that was mostly people, a drive-thru "voice AI" that offshore staff completed) are owned elsewhere in this program (see Topic 4.1) (see Topic 3.3), so here it is named only as an attribution warning: when an automation's ROI looks too clean, check whether the saving came from the algorithm or from a labor arbitrage the algorithm is taking credit for. The honest question is what the software changed, holding the labor arrangement constant.
Example 4: The counterfactual that was already moving (pattern). Consider an organization that launches an AI writing assistant the same quarter it also ships a new template library, retires a legacy tool, and onboards a faster editing team. Output per writer rises 20 percent. The assistant vendor claims the 20 percent. The honest analyst asks how much of the 20 percent the template library and the team change would have produced alone, and reports the AI's share as the residual above that trend. Without a holdout team or a pre-trend line, the residual is an estimate, so the honest output is a range. This is not a real named company because the lesson is structural: whenever several improvements land together, the last one to arrive tends to claim the whole gain.
Example 5: The usage cost that grew with success (pattern, large-model economics). A team deploys a metered, cloud-hosted large-model feature that customers love. Adoption doubles, then doubles again. Traditional software would celebrate: marginal cost near zero. But large-model inference billed per use behaves differently, so the cost line rises with adoption, and a feature that was net positive at low volume can turn net negative at high volume unless priced or capped. The honest ROI here is not a fixed number at all; it is a curve that depends on volume, and the measurement must state the volume it assumes and the pricing model it runs under. Reporting the low-volume ROI as the system scales is a quiet way to book a loss as a win. A system running on owned infrastructure under a flat license does not face the identical curve, but check the contract's usage tiers and the hardware refresh cost before assuming the flat price stays flat.
Example 6: The successful kill (pattern, and a forward pointer). Not every honest ROI is positive, and the most valuable honest measurements are sometimes the ones that justify stopping. A public example of a deployed automated system withdrawn once its true effects were seen is owned as a full case in Topic 8.3 (see Topic 8.3). The transferable point for this topic: an honest ROI measurement that comes back negative is not a failure of the analysis. It is the analysis doing its job, and it is worth more than an optimistic number that keeps a value-destroying system alive.
Example 7: The window chosen to flatter (pattern, timing). An organization reports the ROI of a new AI feature after its first month: strongly positive. The champion presents it as the steady-state return. But the first month captured the launch benefit before the supervision team had met the hard cases, before inference cost grew with adoption, and before any drift set in. Measured over a full year, the same feature is close to break-even. Neither number is fabricated; the dishonesty is in presenting a short-window snapshot as the durable return without stating the window. The honest version reports the month-one figure explicitly as month one, names which costs the window has not yet captured, and projects a range for the annualized number. This is structural, not tied to one company, because the temptation to measure early is universal.
Example 8: The cost that only moved (pattern, cost shifting). A content team deploys an AI drafting tool and reports that drafting time per document fell by half, a clear team-level saving. What the team's ledger does not show is that the editing team downstream now spends more time correcting AI drafts that look polished but contain subtle errors, so the organization's total cost to produce a correct document is flat or higher. The producing team booked a real saving; the organization saved nothing. The honest analyst widens the ROI boundary to include the editing team and reports the net effect across both, catching a cost that was shifted rather than removed. Whenever a saving appears at a team boundary, check the team on the other side of that boundary before crediting it.
Example 9: The revenue that the market gave, not the AI (pattern, attribution). A company adds an AI recommendation feature and sees revenue per customer rise over the following two quarters, which the vendor credits entirely to the feature. But the same two quarters saw the whole market grow, a seasonal peak, and a separate pricing change. Without a holdout set of customers who did not receive the recommendations, the AI's share of the revenue gain is unknowable as a point and must be reported as a range, with the low end crediting the market and pricing moves and the high end crediting the feature. Revenue claims are especially prone to this because revenue has many drivers and the AI is only the newest; the honest analyst never lets the newest cause claim a gain the market handed over.
Example 10: The quality that was measured on the wrong axis (pattern, benefit type). An organization reports that its AI system improved "accuracy" on a named metric and books that as a quality benefit. On inspection, the metric it improved was the one the system was tuned to optimize, while an unmeasured dimension of quality (tone, edge-case handling, or fairness across groups) quietly degraded. The headline accuracy number is real; the quality claim built on it is not, because quality is multi-dimensional and only one axis was watched. This is the signature failure of quality claims from the benefit-type table in Section 3C: a cherry-picked metric improves while the quality that was not instrumented falls. The honest measurement asks which dimensions of quality were not measured before crediting a quality gain.
Where people go wrong
- "The vendor's number is a measurement." It is a sales artifact built to win a decision. Treat any figure you did not compute yourself as a claim to be tested at its four joints (benefit, baseline, attribution, costs), not as evidence. This is the correct prior, not cynicism.
- "Before versus after is the right comparison." The right comparison is the world with the AI versus the counterfactual world you would have had without it over the same period. Some of any before-to-after change would have happened anyway. Crediting the AI with the whole change is the single most common way an honest gain is inflated.
- "Activity is value." Tickets closed, messages handled, documents generated, code accepted, and leads scored are activity metrics. They become value only if the underlying outcome actually occurred. A system optimized to produce activity will look excellent while destroying value, which is the deflection trap.
- "Gross benefit is the ROI." A benefit is only real net of the total cost of ownership, and the TCO of an AI system is dominated by costs below the sticker price: supervision, integration, monitoring, inference, incident risk, and transition. Report gross and you have lied by omission.
- "A single number is the honest answer." A point estimate hides its own uncertainty. When attribution and counterfactual are estimated rather than isolated, the honest output is a range with a stated confidence. A defended range is more trustworthy than a bare point, not less.
- "Resolution time proves resolution." Speed of closing a contact says nothing about whether the problem was solved. A three-second false resolution can be worse than a two-hour real one. Always pull the downstream outcome (repeat contacts, satisfaction, churn, refunds) behind any "resolved" metric.
- "Risk does not belong in an ROI number." Two systems with the same expected return but different tail risk are not equally valuable. An honest ROI is risk-adjusted: it names what could go wrong to erase the return and prices the tail, rather than reporting only the sunny mean.
- "The inference cost is a rounding error." For large-model features, per-use cost scales with adoption, so success raises cost. A feature that is net positive at low volume can go net negative at scale. State the volume your ROI assumes; do not report the pilot's economics as the system's economics.
- "Labor arbitrage is AI ROI." If the saving came from moving work to cheaper humans, or from the "AI" quietly relying on humans, then the software did not produce it. Hold the labor arrangement constant and measure only what the software changed. (see Topic 4.1)
- "Honest ROI is about being pessimistic." It is about being right, in either direction. An honest measurement can justify killing a value-destroying system, which is worth more than an optimistic number that keeps it alive. The discipline is structured attribution, not gloom.
- "The window does not matter as long as the numbers are real." When you measure decides what you see. A one-month window captures the launch benefit and almost none of the costs that arrive later; a one-year window captures both. Report a short-window snapshot as the steady-state return and you mislead without inventing a single false figure. Always state the window, the payback status, and whether the benefit is assumed to decay.
- "A saving on my team's ledger is a saving for the organization." Not if the cost was shifted rather than removed. An AI that saves the producing team by burdening the reviewing team, or saves your department by degrading the customer's experience, books a cost transfer as a saving. Draw the ROI boundary wide enough to catch the cost you may be exporting, and say so when a saving is really a transfer.
- "Payback proves the system is good." Passing the payback point tells you cumulative benefit has overtaken cumulative cost to date; it does not tell you the steady-state return or that the benefit will hold as the model drifts. A just-past-payback number is an early reading, not a verdict. State which side of payback you are on and what the return looks like after it.
Questions people ask
- What is return on investment (ROI)?
- The value a decision produces set against what it cost, usually expressed as a ratio or a net figure over a defined period. Honest ROI is the value genuinely attributable to the AI, against a real baseline, net of the full cost of ownership. More on Return on investment (ROI)
- What is sales artifact?
- A number or claim constructed to win a decision rather than to measure reality. Vendor decks and internal launch slides are sales artifacts; they are the input to an honest measurement, never a substitute for one.
- What is baseline?
- The state of the world a benefit is measured against. The most manipulated part of an ROI claim, because whoever chooses the baseline controls the apparent size of the gain. More on Baseline
- What is counterfactual?
- The world you would have had without the AI over the same period. Honest ROI is the difference between the real world and the counterfactual, not between past and present. Some of any before-to-after change would have happened anyway. More on Counterfactual
- What is attribution?
- The claim that the AI specifically caused a benefit. Attribution fails when a benefit with many simultaneous causes (layoffs, product changes, seasonality) is credited entirely to the AI. More on Attribution
Keep going
This lesson builds ROI and value measurement for AI, and that page shows the roles that hire for it. Every Certified AI Governance Professional (CAIGP) lesson.
Watch the lesson and prove it
Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.
Watch the lesson and prove it →