A/B test
An experiment that randomly splits a population into a treatment group (gets the AI) and a control group (does not), so that because the groups are alike on average, any outcome difference is attributable to the AI. Requires a pre-set sample size, a clean uncontaminated control, a fixed end date with no peeking-and-stopping, and a pre-declared win threshold.
Defined in 2 GAGE programs, which carry 2 distinct definitions of it. The wording above is taught in Certified AI Transformation Professional (CATP).
How each discipline defines it
The same term does different work depending on who is using it. These are the definitions as each program teaches them, unedited.
An experiment that randomly splits a population into a treatment group (gets the AI) and a control group (does not), so that because the groups are alike on average, any outcome difference is attributable to the AI. Requires a pre-set sample size, a clean uncontaminated control, a fixed end date with no peeking-and-stopping, and a pre-declared win threshold.
Running two versions of a system on separate slices of live traffic to compare their real-world behavior. A refinement step for a change that already passed offline testing, never a substitute for the pre-ship regression battery on risky changes.
Where it is taught
The exact lessons this term appears in. The first module of every program is free with a free account.
- Fixing the model, breaking it again: why fixes are never free · Build Before You Govern, Certified AI Governance Professional (CAIGP)
- The AI Value Office and Experimentation Discipline · Implementation from Pilot to Scale, Certified AI Transformation Professional (CATP)
Where it shows up at work
A definition is where A/B test starts. The lessons above teach it inside the competencies below, which are what job descriptions ask for and what the GAGE credential records. Every link here is drawn from the same careers map that builds a role's readiness check.
Builds an honest value model: baseline, measured change, cost, risk, and the projects that should be stopped.
One lesson on this page teach it as a primary outcome.
Decides where AI should be used, where it should not, and which use cases create measurable value, tied to the organization's priorities.
One lesson on this page teach it as a primary outcome.
Roles that hire for it
Each role page shows the full competency set, what the seat decides, and a readiness check against your own evidence.
- Chief AI Officerneeds it at depth
- Chief Information Officer, technology leadership editionneeds it at depth
- AI Adoption and Enablement Leadneeds working knowledge
- AI Program Managerneeds working knowledge · 11 open postings this week
- Operations Manager, AI-readyneeds working knowledge
- Chief Technology Officer, AI engineering editionneeds working knowledge
How job descriptions say it
Postings rarely use the textbook word. These are the phrases the careers map reads as asking for the competencies above.
One interview question about it
How would you measure the return on an AI deployment honestly, including the costs people prefer to leave out?
Asked of candidates for roi and value measurement for ai. Answering it in your own words is exactly how a GAGE topic is passed, and the credential records the pass where an employer can check it.
If you come from another field
Some backgrounds already carry this competency, and the readiness check gives credit for it.
- A business, MBA or technology management degree carries it as a claim to back up: Value and return measurement are a course you passed.
Terms it appears with
Not an alphabetical neighbourhood: these are the terms taught in the same lessons, ranked by how often they appear together.