Skip to main content

Operating-Effectiveness Testing: The Sample, the Exception, the Finding

The short answer

Design effectiveness and operating effectiveness are two different claims

A control can be well designed and still have never actually operated, on the population it claims to cover, the way its description implies. Only real evidence pulled from a real period answers the second question.

What you will be able to do

  • Distinguish design effectiveness from operating effectiveness, explaining why a control that survives design review can still fail a live test, and why the two are separate claims requiring separate evidence.
  • Define a population for a control test precisely enough that a stranger could pull the same set of items, and verify that population's completeness before trusting anything sampled from it.
  • Select a sample using a defensible method, random, judgmental, or a documented combination, and size that sample against a stated tolerable deviation rate rather than a guess.
  • Pull evidence for every sample item and classify each result as a pass, a true exception, an explainable anomaly, or a scope exception, using consistent, stated rules rather than case by case judgment calls.
  • Reach an operating-effectiveness conclusion by comparing the exception rate the sample actually produced against the pass condition set in advance, and explain what the conclusion does and does not prove about the rest of the population.
  • Write a complete finding using condition, criteria, cause, effect, and recommendation, and classify its root cause into one of two families, an execution gap or a design or scope drift, naming a remediation owner distinct from the control's usual owner.
  • Analyze a real regulatory failure, the TD Bank Bank Secrecy Act case, to see exactly how an untested population produced a decade long, undetected control failure, and connect that failure to the exact step in this topic's method that would have caught it.

The ideas, one by one

The population is where most real tests actually succeed or fail

A sample pulled from the wrong population proves nothing about the right one, no matter how careful every later step is; the TD Bank case shows a decade long gap that a single completeness check would have surfaced immediately.

Confirm the population against an independent source before you sample

A population handed over by the control's own owner, unreconciled against a general ledger, an audit log, or an HR record, can silently exclude exactly the items most likely to fail.

Choose and state a defensible sampling method: random, judgmental, or a documented combination

Haphazard selection dressed up afterward as either is not a defensible method, and a workpaper should say plainly which method was actually used.

Set the sample size and the tolerable deviation rate together, before you see any results

The lower the tolerable deviation rate, the larger the sample needs to be to give real confidence a true problem would show up in what was actually examined.

Pull the actual evidence for every sample item, not a summary of it

A test built on a control owner's verbal assurance instead of the underlying artifact has recorded a claim, not run a test.

Classify results with consistent rules: pass, true exception, explainable anomaly, or scope exception

An anomaly classification needs its own independent evidence, and a scope exception calls the whole population into question rather than counting as one more failed item.

A finding needs all five parts, condition, criteria, cause, effect, and recommendation, or it is not usable

A finding that stops at condition and skips cause hands the reader a problem with no path to a fix.

Root causes sort into two families, an execution gap and a design or scope drift, and the right fix depends on which

The TD Bank case carried both at once, understaffed alert review layered on top of a monitoring population that had silently shrunk to 8 percent of actual volume; applying only one family's fix leaves the other wide open.

A clean result is a data point about the tested population, not a permanent verdict about the control

Retest on the control's stated frequency, and treat a first ever clean result on a control that has run unchecked for years as a prompt to double check the population before celebrating it.

Judgmental sampling is the stronger choice, not a shortcut, when risk genuinely concentrates

A random sample on a population where a small subset carries most of the exposure can miss that subset entirely by chance; a stated, explicit risk criterion targets exactly where a real problem is most likely to appear.

Write the honest conclusion down, whatever it is

Every earlier step in this method can be executed perfectly and still be undermined by a tester who softens an inconvenient result under organizational pressure; the discipline exists specifically to remove that discretion at the moment it matters most.

Listen to it

The same lesson, talked through by two hosts as episode 83 of the podcast.

Real cases

These are documented cases and instruments used to illustrate operating-effectiveness testing, not to predict your organization. Each is cited and used for a specific point.

Example 1: United States v. TD Bank, N.A., the population that was never actually tested (this topic's anchor). TD Bank pleaded guilty on 10 October 2024 to Bank Secrecy Act and money laundering conspiracy violations, the largest bank in United States history to do so, after federal investigators found the bank's automated transaction monitoring system excluded domestic ACH transactions, most check activity, and other transaction types, leaving 92 percent of total transaction volume, roughly 18.3 trillion dollars, unmonitored from 2018 to 2024, with the system's rules left substantively unchanged from at least 2014 through 2022. (DOJ press release, 10 October 2024; DOJ Statement of Facts, 2024.) The value for this topic is exact: the government's investigation was, functionally, a population definition and completeness test the bank had never run on its own control, and it took less than one direct comparison, the system's covered population against the bank's actual transaction volume, to expose a decade long gap.

Example 2: FinCEN's findings on unresolved alerts, an execution gap layered on the same case. FinCEN's coordinated action found the bank willfully failed to file Suspicious Activity Reports on thousands of transactions totaling roughly 1.5 billion dollars, and separately documented a 2021 case in which a bank employee facilitated money laundering for narcotics proceeds in exchange for bribes, opening accounts for shell companies conducting funnel account activity in high risk jurisdictions. (FinCEN press release, 10 October 2024.) The value for this topic is distinct from Example 1: this is the execution gap half of the case, alerts and reviews that should have happened and did not, evidence this topic's step six classification, true exception versus scope exception, is specifically built to separate from the population failure Example 1 illustrates.

Example 3: The professional auditing standard behind design versus operating effectiveness. Long established internal control auditing practice draws a sharp, named distinction between evaluating whether a control's design would work if performed as described and evaluating whether the control actually operated as designed throughout a real period, requiring walkthroughs and evidence of actual operation rather than a review of the control's documentation alone. The example is cited here only for its terminology and its general, well established distinction, not as a deep case study; it is the same distinction this topic's own 3B draws directly, expressed in a different professional vocabulary.

Example 4: The EU AI Act's post market monitoring duty presumes an operating test, not a design review. Article 72 of Regulation (EU) 2024/1689 requires providers of high risk AI systems to actively and systematically collect and analyze real performance data after deployment (Regulation (EU) 2024/1689, Article 72; established). The example matters because it shows a legal duty, not merely good practice, presumes exactly this topic's distinction: a design that looks sound on paper discharges nothing under Article 72 without ongoing evidence the system actually performed as claimed once deployed. The Digital Omnibus deferral (Regulation (EU) 2026/1744) pushed the compliance date for this and the other high risk obligations to 2 December 2027 for stand alone Annex III systems and 2 August 2028 for Annex I embedded systems, so the duty is not yet enforceable for most systems as of this writing; the underlying demand it will impose is unchanged.

Example 5: ISO/IEC 42001's internal audit function requires evidence of operation, not description. The certifiable AI management system standard's internal audit clause requires evaluating whether stated controls are actually being followed and are effective, which is answerable only with real evidence pulled from a real population, not a reading of the control's written description (ISO/IEC 42001:2023; established, voluntary standard). The example reinforces that even a management systems standard, independent of any single incident, drives the same population and evidence discipline this topic teaches.

Example 6: The TD Bank remediation structure, matched to two root cause families. FinCEN's four year monitorship for TD Bank requires, among other elements, an accountability review assessing personnel and escalation failures alongside a separate data governance review aimed at the program's structural gaps (FinCEN press release, 10 October 2024). The value for this topic is direct: the regulator's own remediation design mirrors this topic's two root cause families, an accountability review targeting the execution gap and a data governance review targeting the design and scope drift, rather than treating the case as a single, undifferentiated failure needing one fix.

Example 7: PCAOB inspection findings, the regulator that inspects the inspectors. The Public Company Accounting Oversight Board's own inspection reports repeatedly cite insufficient testing of controls' operating effectiveness, including undersized samples and testing that does not establish a control actually operated with enough precision to catch a real problem, as among the most common deficiencies found across audit firms (PCAOB inspection reports and staff analyses of inspection results; established, recurring finding). The point for this topic is structural: the failure this topic's method targets, an under supported effectiveness conclusion, is not rare or exotic; it is the most common way a real control test quietly falls short of proving what it claims to prove, confirmed by the one body whose entire job is checking whether testers actually tested enough.

Where people go wrong

  • Treating a passed design review as proof the control operates effectively. Design effectiveness and operating effectiveness are separate claims requiring separate evidence; a control can be well designed and still have never actually run as described on the population it claims to cover.
  • Accepting the control owner's stated population without an independent completeness check. A population handed over without reconciliation against an independent source, a general ledger, an audit log, an HR system, can silently exclude exactly the items most likely to fail, the way TD Bank's monitoring population excluded 92 percent of total transaction volume for years.
  • Confusing a clean monthly summary report with a real test. A report that reliably arrives and shows a good number is evidence the reporting mechanism operates, not evidence the underlying population it summarizes is the right one; always ask what population the report is actually describing before trusting its number.
  • Sampling without stating the selection method. Haphazard selection, "whatever came up first," dressed up afterward as if it were random or judgmental, produces a sample that cannot support the conclusion drawn from it, and a workpaper should name the actual method used, including when that method was not rigorous.
  • Setting the tolerable deviation rate after seeing how many exceptions turned up. A threshold decided once the results are already known is a story fit to the data, not a test, echoing exactly the pass condition discipline Topic 10.7 already teaches for a control's own test procedure.
  • Classifying a borderline result as an explainable anomaly without independent supporting evidence. An anomaly classification needs its own documented, legitimate reason; classifying too many uncomfortable results as anomalies without that evidence is a quiet way of loosening the test without technically changing the tolerable deviation rate.
  • Treating a scope exception as just one more failed item. A scope exception calls the population itself into question and often invalidates the whole test, not merely the one item that revealed it; folding it into the exception count as though it were an ordinary true exception understates how serious the finding actually is.
  • Writing a finding that stops at condition and skips cause. A finding stating only what was found, without the actual mechanism behind it, hands the reader a problem with no path to a fix; go find the real reason, not just restate the result in different words.
  • Naming a root cause family without checking whether both are present. Some failures, including the anchor case, carry both an execution gap and a design or scope drift at once; fixing only the family that is easiest to address, usually the execution gap, leaves the other one, often the more consequential one, completely open.
  • Assuming a large, well resourced organization would never miss something this basic. TD Bank was, by size and resources, a sophisticated institution, and the gap persisted for roughly six years anyway, because nobody performed the specific completeness check this topic teaches, not because the failure required unusual sophistication to produce or to hide.
  • Skipping the retest after remediation, or running it too soon. A finding closed out on the strength of a fix being implemented, without a follow up test confirming the fix actually closed the population or execution gap, leaves the organization trusting an unverified repair exactly the way it once trusted an untested control; a retest run before the remediated control has completed even one full operating cycle confirms only that the fix was deployed, not that it works.
  • Letting the tester who wrote the population definition also be the only one who checks its completeness. The same structural pressure toward a favorable read that Topic 10.7 named for a control's day to day owner applies to population definition and completeness checking; wherever possible, have a different person perform or review the reconciliation.
  • Reaching for random sampling by default even when risk is concentrated in a small subset of the population. A random sample can miss the highest exposure items entirely by chance; where risk genuinely concentrates, a stated judgmental criterion targeting that concentration is the stronger, more defensible choice.
  • Softening an inconvenient conclusion because the organization does not want to hear it. The exception rate and the effectiveness conclusion follow directly from the classified evidence and the threshold fixed in advance; write the honest number down, whatever it is, rather than easing the language to make an uncomfortable result read better.

Questions people ask

What is operating effectiveness?
Whether a control, as actually built and actually run, performed as designed over a real period of time, on a real population, supported by real evidence a reviewer independently examined. A separate claim from design effectiveness, requiring its own evidence.
What is design effectiveness?
Whether a control, if performed exactly as described, would actually catch the problem it claims to catch. A control can pass this test on paper and still fail an operating-effectiveness test entirely.
What is population?
The complete set of items a control was supposed to act on during the period being tested, defined by its universe, period, source system, and any stated, reasoned exclusions.
What is completeness check?
Reconciling a stated population against an independent source, a general ledger, an audit log, an HR record, before trusting it enough to sample from. The single step that would have surfaced the TD Bank case's population gap years earlier.
What is sample?
A subset of the population actually examined, selected by a stated, defensible method, random, judgmental, or a documented combination, sized against a tolerable deviation rate fixed before the test begins.

Keep going

Watch the lesson and prove it

Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.

Watch the lesson and prove it →