Calibration

The property of a system's confidence scores accurately reflecting its actual reliability, such that a stated ninety percent confidence corresponds to being correct roughly ninety percent of the time on that class of input. A well-calibrated system is the engineering target this topic argues for, rather than simply a maximally cautious one.

The property that a system's stated confidence matches its real accuracy, so that instances it reports as 90 percent confident are correct about 90 percent of the time on your data. A confidence threshold is only a safe boundary if the confidence is calibrated; an uncalibrated model routes confident errors past the human.

The degree to which a forecaster's stated confidence matches reality, so that of all things called 70 percent likely, about 70 percent occur. The defining trait of Tetlock's superforecasters and the real test of a forecaster, human or machine, above raw accuracy on any single call.

The property of a confidence score being honest: among predictions the model reports at a given confidence, it is correct at roughly that rate. An overconfident model (says 90, right 70) is dangerous because users trust predictions they should question.

The process of characterizing and correcting systematic sensor errors (bias, scale factor, misalignment, distortion) by measuring the sensor against a known ground truth under controlled conditions.