A/B test
An experiment that randomly splits a population into a treatment group (gets the AI) and a control group (does not), so that because the groups are alike on average, any outcome difference is attributable to the AI. Requires a pre-set sample size, a clean uncontaminated control, a fixed end date with no peeking-and-stopping, and a pre-declared win threshold.
Defined in 2 GAGE programs, which carry 2 distinct definitions of it. The wording above is taught in Business AI Transformation.
How each discipline defines it
The same term does different work depending on who is using it. These are the definitions as each program teaches them, unedited.
An experiment that randomly splits a population into a treatment group (gets the AI) and a control group (does not), so that because the groups are alike on average, any outcome difference is attributable to the AI. Requires a pre-set sample size, a clean uncontaminated control, a fixed end date with no peeking-and-stopping, and a pre-declared win threshold.
Running two versions of a system on separate slices of live traffic to compare their real-world behavior. A refinement step for a change that already passed offline testing, never a substitute for the pre-ship regression battery on risky changes.
Where it is taught
The exact lessons this term appears in. The first 7 topics of every program are free with a free account.
- Fixing the model, breaking it again: why fixes are never free · Build Before You Govern, AI Governance: Applied Mastery
- The AI Value Office and Experimentation Discipline · Implementation from Pilot to Scale, Business AI Transformation
Terms it appears with
Not an alphabetical neighbourhood: these are the terms taught in the same lessons, ranked by how often they appear together.