Designing a Control That Can Be Tested

October 2022. A high school student walks through the front doors of their school and they pass directly through an AI-powered security scanner. Right.

And this, I mean, this isn't some legacy metal detector from the 90s. This is a system marketed as the absolute cutting edge of artificial intelligence. It's utilizing advanced sensor fusion, object classification models, the whole nine yards.

So that's supposed to be the gold standard. Exactly. But the student is carrying a seven-inch knife and the AI scanner, it does not flag it.

Doesn't even beep. Doesn't beep, doesn't light up. The student just walks right through.

And tragically, that knife is later used to stab another student. It is, I mean, it's a devastating event. And when you look at it purely from an engineering standpoint, you might just conclude that the tragedy stems from a simple sensor failure.

Right. Like the machine just broke that day. Exactly.

But if you're a professional, you know, responsible for AI systems, and you're looking at this through the lens of risk management, this sensor failing is actually not the core issue here. Wait, really? How is the sensor failing not the issue? Because sensors fail. I mean, they just do.

AI object recognition models, they suffer from occlusion, they misclassify edge cases, they degrade over time. That's a given. A mature governance program does not operate on this delusion that a sensor will perform perfectly forever.

Ah, okay. So it's about the backup plan. Right.

It engineers safety nets for the inevitable moment the sensor is wrong. Which brings us to the actual governance failure, right? The one that triggered a massive regulatory hammer. Because when the Federal Trade Commission, the FTC, opened the hood on this company Evolve Technologies in their November 2024 complaint, they didn't just find a broken machine.

No, they found something much more systemic. Right. They found a company making claims about their AI that were fundamentally, like, completely untestable.

I mean, Evolve told its customers, which were primarily K-12 schools, that the Evolve Express system, quote, detects all weapons. It detects all weapons. Yeah.

That is the phrase that should immediately, like, set off alarm bells for anyone who has ever built an AI model. Because all is a very strong word in AI. It's an impossible word in AI.

Yeah. But they went further. They promised it ignores harmless personal items, like laptops and water bottles.

So kids didn't have to empty their backpacks, which is a huge selling point. A massive selling point. And they claimed this magical accuracy would allow schools to cut security labor costs by, like, 70 percent.

Wow. 70 percent. But in the FTC's complaint, the allegation was that not a single one of those specific, seemingly testable claims was backed by the kind of evidence that would allow an independent party to verify it.

Right. Like a school district or an auditor or a regulator. None of them could actually check if these claims were true.

Evolve basically handed their customers a control for weapons detection accuracy that consisted of absolutely nothing but a marketing sentence. Marketing sentences masquerading as controls. That is exactly it.

So there was no stated mechanism, like no instruction manual for a school administrator to run a test and say, OK, Evolve, your machine failed. Exactly. And the legal fallout from that lack of a mechanism was absolute.

I mean, fast forward to March 2025. A federal judge in Massachusetts approves a stipulated order. Evolve is now legally barred from making unsupported claims about their system's capabilities.

And the schools got it out, too, didn't they? Yeah. K-12 schools that signed contracts based on those claims gained the right to just outright cancel them. Plus, Evolve is locked into ongoing compliance, reporting to the FTC.

So the FTC's action against Evolve, this is really the anchor for our deep dive today. We're going to treat this as an executive education session on how to build AI governance controls that can actually survive a hostile audit. Because the Evolve case, it illustrates the entire thesis perfectly, you know.

Right. If you have a sentence sitting in your governance file that sounds exactly like a control like, our system detects all weapons, but nobody can definitively state what specific sequence of events would have to happen for that statement to be proven false. Then you do not have a control.

You have a sentence. You have a sentence. Which leads us to the fundamental spine rule of this entire deep dive.

And we're going to come back to this a lot. A control has to be able to fail. Or it is not a control.

It is the absolute bedrock of risk management. A control has to be able to fail. Let me just repeat that for you.

Because if you take nothing else away from our time today, it has to be this. A control has to be able to fail, or it is not a control. Amen to that.

But I want to explore the psychology of why that is so hard for organizations to accept. Because in my experience, organizations deceive themselves constantly about this. Oh, all the time.

It's systemic. Right. You walk into a Fortune 500 company, you ask about their AI risk management, and they hand you this beautiful, heavy, perfectly indexed binder of governance documentation.

With the corporate logo embossed on the front. Exactly. It looks incredibly professional.

The formatting is flawless. But when you start reading the actual text under the headers, the mechanisms are just completely untestable. Why do smart executives accept this? Well, they accept it.

Because the vocabulary of governance has been entirely corrupted over the years. Corrupted how? People use words interchangeably that actually have vastly different meanings. To break that corruption, we had to define our terms strictly.

We need to distinguish between three distinct artifacts that always show up in these binders. Okay, lay them out. A good intention, a policy statement, and a true control.

Let's start with the good intention. What is that? A good intention is a stance. It's a corporate commitment that names no specific activity whatsoever.

So like a PR statement, basically. Pretty much. You might see a sentence in the binder that reads, we take model accuracy seriously and are committed to minimizing biased outputs.

I mean, that is just corporate poetry. There is no mechanism there. I can't walk up to a data scientist on a Tuesday and ask them to show me the evidence of taking something seriously.

Exactly. Evolves Public Marketing leaned heavily on this kind of language, you know? Projecting supreme confidence with zero stated mechanism for how that capability was systematically validated. Right.

It's the easiest thing for a committee to write. And it is the least defensible thing you could ever put in front of a regulator. Yeah.

But the second category is much more insidious, I think. Policy statement. The policy statement.

Yeah. This is where I see the most confusion, honestly, because a policy statement wears the camouflage of a control. It really does.

It sounds like one. Let me give you an example that I see in almost every HR tech governance file I review. AI generated hiring recommendations are reviewed by a human before any candidate is rejected.

Classic. Now, if I'm an executive scanning a document, that sounds like a solid control. It names an activity.

Human review. It names a trigger before a candidate is rejected. Why is that just a policy statement? OK, let's view that exact sentence through the lens of a hostile auditor.

The sentence says a human reviews it. Right. Which human? Doesn't mean the hiring manager, the HR director, a third party auditor.

It doesn't say. Furthermore, what is the human actually checking for? Are they checking the model for racial bias or are they just checking for formatting errors in the resume? Right. It doesn't specify the criteria.

Exactly. How often are those reviews sampled to ensure the human isn't just mindlessly clicking approve because they're tired? And here's the absolute kill shot from an auditor. What physical or digital evidence exists that the review happened at all? Oh, yeah.

A policy statement describes a rule the organization wants followed. It is a description of an ideal state. An ideal state.

Yes. It does not describe how anyone inside or outside the company would actually know if the rule was followed on a random Tuesday when everyone is overworked and just rushing to go home. OK, so a good intention is a stance.

A policy statement is a desired rule. What is the real deal? How do we define a true control? A true control is a defined activity performed by a named party triggered by a specific event or schedule producing evidence every time it runs and testable against a stated past condition using a stated sample. Wow.

OK, that is incredibly dense. And we are absolutely going to break that definition down into its component parts in a moment. But first, I want to establish our second core principle for you, the listener.

Yeah, this is vital. A policy statement and a good intention both masquerade as controls, and both are more dangerous than an honest gap. They are so much more dangerous.

It's this psychological trap of compliance work. How so? Think about it. When a team spends three months drafting a comprehensive AI policy, you know, they're arguing over definitions, getting sign off from legal, from engineering, from the C-suite.

It feels like risk reduction. Because they spent a lot of money and time on it. Exactly.

The tone of the final document is highly authoritative, but that confident tone masks a total lack of structural mechanics. See, I'm going to push back on that framing, though, because isn't writing that policy the absolute hardest part of the job? I wouldn't say the hardest. Well, you have to forge alignment across departments that usually hate talking to each other.

Like, getting legal and engineering to agree on anything is a miracle. Building a policy is like drawing the architectural blueprint for a skyscraper. You literally cannot build the building without the blueprint.

Why diminish that effort? The blueprint is necessary, yes. I'm not diminishing the effort of drawing it. But a blueprint does not pour the concrete, right? Right.

And a blueprint certainly doesn't prove the steel won't buckle when a hurricane hits. A blueprint is just paper. You need a structural inspector to go to the physical site, test the concrete, and verify that the physical reality actually matches the drawing.

That makes sense. Let's look at the alternative. An honest gap.

If you're writing your risk register, we currently have no fairness check on our resume screening model. That is a known vulnerability. It's an open wound, but at least you know you're bleeding.

Precisely. Yeah. Track it, allocate budget for it, prioritize engineering time to fix it.

But a policy statement masquerading as a control, that is an unknown vulnerability wearing the disguise of a solved problem. It's an anesthetic. Exactly.

It acts as an anesthetic to executive anxiety. The board reads the policy, they feel safe, and they just stop asking questions. Until the auditor shows up.

Right. When an auditor finally arrives and looks at that filing cabinet of good intentions, all it takes is one simple demand to collapse the entire illusion. Which is? Show me the test.

Show me the test. Okay. So if the blueprint isn't enough, we have to transition from drafting hollow sentences to engineering rigid verifiable mechanisms.

How do we take the theory of a policy and transform it into the steel of a control? We enforce a strict structural framework. If you want to design a control that an auditor cannot break, a control that can actually protect your organization, it must contain five specific elements. Five elements.

Yes. If you skip even one of these, the control cannot run reliably. It cannot be verified or it cannot be defended.

This is our next spine rule, isn't it? It is. Five elements or it is not testable. Owner, trigger, frequency, evidence artifact, test procedure.

Let me anchor that for you. Five elements or it is not testable. Owner, trigger, frequency, evidence artifact, test procedure.

Let's get deeply tactical here and unpack the first four right now. These four are what actually make the control run in the real world. Let's do it.

Element one, owner. The owner must be a specific named role. Not a team, not a committee, and not a department.

So no generic assignments. None. You cannot write, the data science team monitors the outputs for bias.

Hold on. I see that everywhere. And honestly, it makes intuitive sense to me.

Isn't naming a team much more resilient? How so? Well, if I name a specific role, say the senior ML engineer, and that person gets fired or goes on a three-week safari, the control dies with them. If I assign it to the data science team, anyone on the roster can pick up the slack. Doesn't a team assignment ensure the control survives personnel changes? It sounds logical.

I'll give you that. But organizational psychology proves it is a disaster. Assigning ownership to a team diffuses accountability across multiple people.

Meaning nobody owns it. Exactly. It's the bystander effect applied to corporate governance.

Everyone assumes someone else on the team ran the bias check this week. Oh, I thought Dave was doing it. I thought Sarah did it.

Exactly. When the control inevitably lapses and an auditor sits across the table asking who specifically is responsible for this failure, handing them a department name is an admission that the control had no real owner. So how do you handle the reality of turnover, though? What happens when the role is vacant and there's a corporate hiring freeze? You assign it to a role like data governance lead.

If the human in that role leaves, standard operating procedure dictates that the responsibilities of the vacant role roll up to their immediate supervisor, say the VP of engineering, until the role is filled or officially reassigned. OK, so the accountability travels up the org chart. Yes, but it remains singular and inescapable at all times.

That makes sense. Singular accountability. If a failure occurs, we know exactly whose office to walk into.

Let's move to element two, trigger. The trigger is the exact unmissable event or specific schedule that starts the control. It can be time based, such as on the first business day of every month at 9 a.m., or it can be event based, such as every time a new training data set is uploaded to the S3 bucket.

Or like every time the model architecture is modified. And if you don't define a trigger, like if it just says we check for bias. But a control has no stated trigger.

It runs whenever someone happens to remember it. And in corporate reality, amidst quarter-end pushes and sprint deadlines, things that run when remembered run rarely, unpredictably, and usually only after a catastrophe has already happened. Right.

They only remember it when legal is yelling at them, which flows naturally into element three, frequency. Yes, how often the control repeats. Monthly, quarterly, continuously, on every single deployment.

Why is frequency so critical to define separately from the trigger? Because frequency is the dividing line between a living governance program and a graveyard of past achievements. A graveyard of past achievements. I like that.

Think about it. Consider an AI model that was rigorously tested for security vulnerabilities right before its initial launch. Okay.

The team did a great job, documented everything, and deployed it. If they never revisit that test, that is not a control. That is an expired control.

Because the model changes. Exactly. As the model encounters new real-world data, as its dependencies update, as the underlying architecture shifts, the risk profile changes entirely.

So a control is not a one-time gate. It's an ongoing pulse. Absolutely.

Now, let's look at the fourth element, which is the one that bridges the gap between the physical world and the digital world, the evidence artifact. This is critical. The evidence artifact is the specific named output the control produces every single time it successfully runs.

It is the footprint. It is the footprint. A cryptographic hash, a signed log, a time-stamped approval field in JIRA, an exported PDF data report.

If this artifact is not physically or digitally produced, your risk register has nothing to point to. It's just a rumor. Literally.

Your governance reverts to an oral tradition, an assertion that we definitely did it. I want to spend some time here because the lack of an evidence artifact is often the fatal flaw in massive corporate disasters. Let's look at the Night Capital trading disaster from August 2012.

Oh, this is the ultimate case study. Right. It wasn't an AI failure, but it is the perfect example of deployment controls failing.

Night Capital was deploying new routing software called SMARS to their trading servers. Right, a routine update. What they didn't realize was that an old, dead piece of code, a routing flag called PowerPeg that hadn't been used in nearly a decade, was still sitting dormant on one of their eight servers.

Just the eighth server. Ticking time bombs. Exactly.

When they flipped the switch on the new code, they inadvertently reactivated PowerPeg on that eighth server. And the results were apocalyptic. Completely apocalyptic.

The zombie code started buying high and selling low at algorithmic speed. In 45 minutes, Night Capital lost $460 million. Just staggering.

They went effectively bankrupt in less than an hour. Now, if you looked at Night Capital as paperwork, they absolutely had deployment controls. They had policies dictating how software should be deployed across their servers.

But what they lacked was the artifact. Exactly. Let's apply our elements.

They had a trigger the software update. Right. But they lacked an evidence artifact proving the state of the servers before the trigger was pulled.

If their control had required a specific artifact, say, an automated time-stamped script that cross-referenced the active code on all eight servers against the approved master branch producing a signed log file, the discrepancy on the eighth server would have been caught instantly. Because the policy said, deploy correctly. The missing artifact allowed them to deploy incorrectly without anyone noticing until the market opened.

They had a control on paper, but it lacked the physical trace to prove the operation was actually executed correctly. And that brings us to a massive realization about these first four elements. The owner, the trigger, the frequency, and the evidence artifact.

They make a control operation. Right. They make it run.

They guarantee that an activity happens and leaves a trace, but they do absolutely nothing to prove that the activity was correct or effective. That's a vital distinction. You could have a monthly data quality report that triggers reliably on the first of the month, gets emailed to the data governance lead, and gets dutifully filed away in a designated folder.

The first four elements are perfectly satisfied. But what if the data in that report is completely corrupted? Exactly. The artifact exists, but it's garbage.

Evidence that nobody rigorously checks is just a highly organized good intention, which is why the first four elements are only the vehicle. To actually drive risk down, you need an engine. And that brings us to the fifth element, the absolute core of verification, the test procedure.

The test procedure, yes. In my experience, when you look at an average corporate governance file, this is almost entirely absent. The file lists the owner and the artifact, and then it just stops.

Because writing a test procedure requires committing to a standard of failure, and people hate doing that. It's scary. It is.

The test procedure is the exact step-by-step method that a reviewer, ideally someone structurally independent from the controls owner, follows to verify that the control is actually working as intended. And here is the spine rule you must adhere to, for you listening. The test procedure needs two specific parts, a sample size and a pass condition.

Crucial. A sample size and a pass condition. Let's dissect those, starting with the sample size.

What makes a sample size defensible to an auditor? A defensible sample size must be a fixed count, a strict percentage, or the full population if the volume is low enough. So specific numbers. Always.

You must write, review 25 randomly selected auto rejections, or sample 5% of all lending decisions made this month. What you absolutely cannot use are the vague placeholders that infect most compliance documents, like check a few outputs or review a representative sample. I have to stop you on representative sample, because if I'm talking to a data scientist, representative sample is a highly technical, rigorous concept in statistics.

Why is that phrase so dangerous in a governance control? It sounds rigorous, right? But in the context of a written control procedure, it is usually a placeholder for a decision that the author was too lazy to make. Really? Just laziness. Or avoidance.

Imagine you hand a test procedure that says, Check a representative sample to two different independent reviewers. Reviewer A might decide that out of a million transactions, three is representative enough if they cover different geographies. Right.

Reviewer B might decide they need to pull 3,000 transactions to achieve actual statistical significance. They are going to pull vastly different amounts of evidence, apply entirely different levels of scrutiny, and almost certainly reach different verdicts on the same control. Which means the test procedure itself is inconsistent.

Exactly. A test procedure that yields different results depending on who reads it is completely useless. The sample size must be explicitly quantified so a total stranger could execute it without having to ask you to clarify what you meant.

Okay, that makes perfect sense. Quantify the sample so the execution is uniform. What about the second part, the pass condition? The pass condition is the razor edge of the control.

It must be stated as a hard number or a specific checkable state. Give me an example. For example, 98% of the sample decisions show a documented human review comment in the jury ticket, or the false positive rate of the model on the test set is below 2%.

Very binary. It either is or it isn't. Exactly.

You cannot use subjective phrasing like results look reasonable or performance is generally accurate or no major anomalies detected. I see results look reasonable all the time. It's not a standard.

It is a shrug wearing the grammar of a standard. That is precisely what it is. It's a shrug.

And this leads us directly to the golden rule of testing, a spine rule that, if violated, destroys the integrity of your entire risk program. Which is? Set the pass condition before you see the results. Set the pass condition before you see the results.

Let's really dig into the psychology of this because I see brilliant engineers and executives fall into this trap constantly. Oh, it happens every day. When you pull the data first and then decide what a passing grade is, you aren't testing the system.

You are storytelling. It is the most pervasive form of corporate self-deception, and it is usually done with the best of intentions, honestly. Walk me through how it happens.

Imagine a team deploys a new underwriting model. They let it run for a month, then pull the logs. They notice that the model flagged 85% of edge cases correctly for human review.

OK, 85%. They look at that 85% and say, you know what? That's pretty good for a first iteration. So they go into the governance document and they write a pass condition.

The control passes if greater than 80% of edge cases are flagged for human review. Wow, they retrofitted the narrative. Exactly.

They have completely removed the reviewer's judgment about whether a failure actually occurred and replaced it with a post hoc rationalization. They looked at what the system happened to do and decided that whatever it did must be the definition of success. It's the equivalent of shooting an arrow into a blank wall, walking up to the wall, and carefully painting a bullseye around exactly where the arrow happened to land.

You get to call yourself a sharpshooter every single time. That is a perfect analogy. And auditors know exactly how to spot the painted bullseye.

How do they spot it? The verdict must be entirely independent of the specific evidence pulled. If an auditor or a regulator looks at your test results, they're going to look at the date the threshold was established versus the date the data was analyzed. They are checking to see if you engineered a test or if you just engineered an alibi.

Engineer an alibi. That's powerful. This is heavy theory.

So let's take all of it, the five elements, the quantified sample size, the preset pass condition, and apply it to the real world. Let's go back to our starting point. Evolve Technologies and their AI weapons scanner.

Let's do it. We know their marketing claim was basically the Evolve Express system detects all weapons and ignores harmless items. If a school district were actually going to deploy the system responsibly, how should they have written that as an actual testable control? Let's rewrite it right now.

Transforming that marketing sentence into a five-element control from the perspective of the deploying school district. Okay. Lay out the first element.

First, the owner. We don't write the school administration. We write the district director of physical security technology.

Singular accountability. Got it. Second element, the trigger.

Initial installation of the hardware and every subsequent software or model update deployed by the vendor. Makes sense. Third, frequency.

Re-verified quarterly and immediately following any triggered update. And fourth, the evidence artifact. A dated signed test report detailing the specific test items presented, the exact detection rate achieved, and the false alarm rate observed stored in the district's compliance portal.

Okay. So those four elements govern the operation. We know who is doing it, when they're doing it, and what paper trail they leave.

Exactly. Now for the engine, the fifth element, the test procedure. How do we write that? The test procedure.

Present a fixed sample set of exactly 50 concealed items to the system. This set must include 20 common school items known to trigger false alarms, like three ring binders and steel water bottles, and 30 weapons, specifically including knives of the dimensions involved in the October 2022 incident. Very specific sample.

The pass condition. The system must successfully flag 100% of the weapons and false alarm on no more than 10% of the harmless items, matching the vendor's required specifications. Any failure to meet these exact thresholds is logged as a critical failure, and the system is taken offline until remediated.

Look at the difference. I mean, just feel the difference in the weight of that language. That rewritten version can actually fail.

Yes. It boldly declares exactly what failure looks like, who is looking for it, and what evidence proves it. The original marketing sentence could never fail because failure was never structurally defined.

And that immense structural distance between the marketing sentence and our five element control is exactly the distance the FTC measured when they filed their complaint. Now, in the complex reality of enterprise AI, you aren't just writing one control for a stanner. You are managing intricate pipelines of data, models, and outputs.

To organize this, we generally break controls down into three essential types based on where they sit in the AI lifecycle. Intake, monitoring, and access. Right.

The three pillars. Let's explore the mechanics of each because they fail in very different ways. Let's start at the beginning of the pipeline, the intake control.

So this governs what is allowed to enter the system. This could be a new raw data set being admitted into a training environment or a new third-party AI tool being approved for employee use. And the evidence artifact here? It is almost always some form of a completed intake checklist or risk assessment form.

Right. But what is the test procedure? If the auditor just checks to see if the form is in the folder, isn't that just a paperwork check? Yes. And that is where the friction of implementation becomes crucial.

If you just check for the existence of the form, you miss the most common failure mode of an intake control. Post hoc paperwork. Data scientists want to move fast.

They find a data set. They start training a model on it. And three weeks later, they remember they were supposed to get it approved by governance.

Ah, so they just quickly fill out the checklist and dump it in the folder. Exactly. They backdate the approval.

Wow. So your test procedure cannot just ask, does the checklist exist? Right. The test procedure must state, compare the timestamp of the completed intake checklist against the first timestamp the data was queried in the development environment.

The control passes only if the checklist timestamp precedes the query timestamp. You design the test to catch the specific human behavior that circumvents the control. Brilliant.

Right. Catch the behavior, not just the paper. What about the second type, the monitoring control? The monitoring control governs systems that are already running in production.

Is the model drifting? Are the outputs becoming biased? Is the latency creeping up? The artifact here is usually an automated periodic report, perhaps a dashboard export showing performance metrics. But dashboards are dangerous. I call it dashboard blindness.

Oh, totally. You have a TV screen in the engineering pit showing 50 graphs, and nobody actually looks at it until something catches fire. An automated report that just gets dumped into an inbox every Friday afternoon isn't a control if nobody acts on it.

Dashboard blindness is a massive risk. Therefore, the test procedure for a monitoring control cannot just check if the report was generated. The test must verify human intervention.

Wow. The procedure should state, Sample 10 weekly performance reports. The control passes if 100% of the sampled reports show a digital signature or long comment from the designated owner within 48 hours of generation, confirming review of the metrics.

So if the machine generated the report, but the human didn't read it, the control has failed. Exactly. The human in the loop is verified.

Finally, the third type, the access control. This governs the perimeter. Who has the privileges to touch the system, retrain the model, or view the sensitive training data? The artifact is pretty straightforward here, right? Yeah, usually an access log or a permissions matrix from the identity management system.

But the failure mode here is notorious. It's the ghost employee. The ghost employee, yes.

Walk me through the mechanics of the ghost employee. Imagine an engineer, let's call him David, works on the core ML model. He has full rewrite access.

Six months later, David transfers to the marketing department to work on ad analytics. Or maybe he leaves the company entirely. Because IT and HR systems are often completely disconnected, David's original access to the core ML model is never revoked.

He is a ghost in the system, holding privileges he no longer needs, creating a massive security vulnerability. So how do you test for that? If I just look at the access list, David's name is on it, and he was legitimately approved six months ago. Which is why the test procedure for an access control cannot just sample the access list against historical approvals.

What should it do instead? The test procedure must state, sample 20 users currently holding read-write access to the model. Cross-reference this sample against the current active HR employee directory and role assignments. The control passes if 100% of the sampled users are actively employed and currently assigned to a role that requires this access.

That is how you catch the ghost employee. That's how you catch it, by checking against the current reality, not the historical paper trail. I want to ground all this theory in a highly realistic, slightly painful scenario.

Let's introduce our listener to Pauline. Pauline is the risk lead at a mid-sized insurance company. Okay, Pauline, let's see what she's dealing with.

The company has recently deployed an AI triage model to scan incoming claims. The model flags obvious claims for fast-track approval and routes the complex or suspicious ones to human adjusters for investigation. Pretty standard use case.

Now, Pauline, as general counsel, forwards her the press release about the FTC suing Evolve Technologies. The GC is terrified and asks Pauline a simple question. Do we have anything like this Evolve marketing garbage sitting in our own governance file? Oh, for a risk lead, that is the nightmare email.

It forces you to look at your own documentation through the unforgiving eyes of a regulator. So Pauline opens her conformity file, scrolls down to the triage model, and finds this sentence under the list of controls. Claims flagged for investigation are reviewed by a qualified adjuster before any denial is finalized.

At first glance, it sounds perfectly adequate. It does. It has an activity review.

It has a trigger before denial. But if Pauline applies our framework, she realizes she cannot answer the general counsel's underlying question. Exactly.

Because if an adjuster is overwhelmed and skips a review to hit their quota, who is accountable? How often is this process audited? What is the physical artifact proving the review happened? What is the past condition for the audit? She realizes she has found her organization's version of the Evolve marketing sentence. But Pauline is a professional. She doesn't panic and she doesn't try to hide it.

She opens a blank template and forces that hollow sentence through the meat grinder of the five elements. Let's watch her work step by step. Let's do it.

First, the owner. She knows better than to write the claims team. She looks at the org chart and assigns it to a specific role, the director of claims operations.

Singular accountability. Perfect. Second, the trigger.

Every time a claim is flagged by the AI triage model. Good. Third, the frequency.

The test of the reviews will be conducted monthly. And fourth, the evidence artifact. And here, Pauline hits a massive operational wall.

Let's explore that wall because this is where governance creates friction with operations. Pauline talks to the IT department to find out how adjusters actually document their reviews. She discovers that adjusters just type their case notes into a generic free text field in the legacy claims system.

There is no distinct artifact. Oh, that's a nightmare. There is no digital signature.

There's no way to definitively prove when the note was written. Right. So IT tells her, yeah, they just type it in the box.

What's the problem? The problem is that a free text field cannot be systematically verified. Pauline realizes that to satisfy the framework, she has to force a change in the software itself. Exactly.

This is the framework forcing architectural improvement. Pauline submits a change request to IT. She demands they build a required review-completed digital checkbox intrinsically tied to the adjuster's login credentials and automatically timestamped by the server.

That's going to annoy the adjusters. Oh, they'll hate it. But furthermore, she requires that the system physically block the issuance of a denial letter until that specific box is checked.

That new, timestamped, cryptographically secure field becomes her evidence artifact. That is going to cause a massive fight with the adjusters, who will hate the extra click, but it is the only way to secure the system. So she has her artifact.

Now she gets the fifth element, the test procedure. Pauline writes, each month, an independent risk analyst pulls a sample of 50 randomly selected denied claims that were originally flagged by the AI. The control passes if 100% of the sample shows a completed review field with the system timestamp that occurs before the timestamp of the denial letter.

Any missing field or any post hoc timestamp, meaning the note was written after the denial went out, counts as a critical failure. And this is the magic of the five elements. Just by writing that test procedure, Pauline has discovered a massive operational gap before an external auditor does.

She realizes that under the current free text system, adjusters routinely type their notes days after the denial letter goes out, just to catch up on paperwork. Right. Her control, as originally designed, would have immediately failed its own test.

It's a brilliant realization. She found the gap by stressing the system on paper. She logs the gap in the risk register.

She initiates the IT fix for the timestamp feature. And then she replies to her general counsel. And she doesn't offer a platitude.

No, she doesn't say we're fine. She says we had a fake control that looked exactly like evolves. We found a real gap in how adjusters document denials, and we were engineering a systemic fix before anyone outside this building finds it.

That is the difference between an organization flying blind and an organization that is actually engineering evidence. It really is. But as we move into the long-term reality of maintaining these systems, we have to acknowledge that even perfectly written controls, even Pauline-esque, can fail in practice.

Designing the control is only the first stage of its life. Which is why we need to transition to the mechanics of scale, edge cases, and how systems break down over time. Because a control that looks perfect on paper must survive contact with a chaotic reality.

And this brings us to another foundational spine rule for you to remember. Match the sample size and frequency to real stakes and real rates of change. Match the sample size and frequency to real stakes and real rates of change.

Let's tackle the stakes and the sample size first. This rule addresses one of the most common failure patterns we see in the field. The mathematically irrelevant sample.

Imagine you have a high-volume AI underwriting process. It's churning out tens of thousands of lending decisions every single quarter. If your test procedure dictates that the reviewer should sample three outputs, your test is entirely worthless.

Because it's a drop in the ocean. Exactly. A sample that small can pass by pure blind luck, even if the underlying system is catastrophically biased and denying loans illegally left and right.

It's the equivalent of flipping a coin three times, getting heads three times in a row, and confidently concluding that the coin does not have a tail side. It is mathematical malpractice. Precisely.

Now, the fix isn't just screaming, pick a bigger number. You have to adjust to the reality of the volume. How so? If you have a low-volume process, say, your system only admits 10 new training data sets a year.

Don't sample at all. Test the whole population. It's cheap, it takes an hour, and it's definitive.

But for high-volume systems, you need someone with basic statistical literacy to calculate a sample size large enough that a real systemic failure rate would actually be detected with a high degree of confidence. So that covers the stakes. But what about the rates of change? How does frequency play into this? This is a second major failure pattern.

A testing frequency completely mismatched to how fast the system is evolving. Give me an example. If your AI model is connected to a dynamic data pipeline, and it is being retrained on new customer behavior every single month, its risk profile is changing every single month.

If your control testing frequency is set to annually, you have engineered a massive blind spot. Because by the time you run your annual test in December, the model you are testing hasn't existed since January. Exactly.

You are verifying a ghost. The frequency of the test must be anchored to the events that fundamentally alter the system's risk profile. If the model updates monthly, the core controls must be tested monthly.

This ties into a broader concept that executives need to grasp. The control lifecycle. Writing the five elements is just the first stage.

Design. Right. The second stage is operate.

The control obviously has to trigger and produce its artifact in the wild. Yep. The third stage is test.

The independent reviewer pulls the sample, executes the procedure, and reaches a verdict. But it's the fourth stage that actually defines the maturity of an organization. The fourth stage is remediate.

Remediation is where the rubber meets the road. Explain remediation. If a test fails.

If polling the analyst finds three claims where the adjuster didn't review the file before denial, you must have a stated response. You need a named owner responsible for the fix, a root cause analysis, and a concrete date to retest the system to confirm the fix actually worked. Here is a crucial insight for everyone listening, and it is a point of immense legal liability.

A failed test with no remediation is a diagnosis with no treatment. It's the worst position you can be in. Let me say that again.

A failed test with no remediation is a diagnosis with no treatment. If you run a test, find a failure, log it in your beautifully formatted register, and then do absolutely nothing about it, you haven't reduced your risk at all. What you have done is create documented, legally discoverable proof that your organization knew about the risk, understood the failure, and consciously chose to do nothing.

It is a prosecutor's dream. It really is. Which is an incredibly dangerous position to be in during an enforcement action, and it leads us to the most corrosive failure pattern of all.

The one that undermines everything we've talked about today. Gaming the test. Gaming the test happens constantly, and it rarely stems from malicious intent.

It happens when the owner of the control is allowed to run their own test. This creates an irreconcilable structural conflict of interest. Because they want a pass.

The owners say the director of claims operations benefits professionally when the control looks good. So unconsciously or otherwise, they pick a sample from a week they know was slow and error-free. Or they set a pass condition so loose that failure is virtually impossible.

Exactly. This is exactly why an unbroken record of passes is a massive red flag, not a trophy. If an auditor comes in, opens your books, and sees that every single control has passed 100% of its tests for three straight years, they aren't going to shake your hand and congratulate you.

No, they're going to immediately assume your testing environment is rigged. Any complex system interacting with unpredictable human behavior and noisy real-world data will produce occasional failures. It has to.

Right. A spotless record usually means you are testing something trivial, or your independent testing is structurally compromised. It would also remove the conflict of interest by ensuring that the person testing the control does not report to the person who owns the control.

I want to take all of this, the framework, the life cycle, the independence, and connect it to the global landscape. Because this isn't just about internal hygiene or making your general counsel happy. We are seeing these exact structural demands being codified into the major regulatory frameworks around the world.

This is the law now. The requirement that controls be genuinely testable and not just stated intentions is explicitly or implicitly embedded in the biggest regulatory instruments governing AI today. Let's look at the European Union's AI Act.

Perfect example. Under Article 72, providers of high-risk AI systems have a strict post-market monitoring obligation. The text requires the active systematic collection and analysis of data regarding the system's performance after deployment.

Active and systematic. You cannot satisfy that language with a policy statement that says we monitor our models. No, you cannot.

Active and systematic analysis requires a defined sample, a defined frequency, and a defined threshold for action. It requires the five elements. If you cannot prove to an EU regulator how you systematically collect and analyze that data, you are in violation of Article 72.

What about the standards side? The frameworks organizations use to build their compliance programs. Look at ISO IE's 42001.2023, the premier certifiable AI management system standard. It strictly mandates an internal audit function to evaluate if your stated AI controls are actually effective.

And you cannot audit effectiveness without a predefined pass condition. Right. If you don't know what failure looks like, an ISO auditor cannot certify your management system.

Or look at the NIST AI risk management framework, specifically the measure function. It calls for tracking AI risks using rigorous quantitative or qualitative techniques. You cannot measure risk with a general commitment to take accuracy seriously.

Exactly. All of these frameworks use slightly different vocabulary, but they're all demanding the exact same architectural reality. Testable, specific evidence.

If you build your controls using the five elements we've detailed today, you are simultaneously preparing yourself for the EU AI Act, the ISO 42001 audits, and the NIST framework. Because they all ultimately boil down to surviving the auditor's attack. The auditor's attack.

I love that framing. Because an audit isn't a friendly conversation. It is an interrogation of your systemic integrity.

It really is. If you understand how a highly trained, genuinely hostile auditor, or a regulatory examiner from the FTC thinks, their attack is highly predictable. When they look at your system, they are going to ask four escalating questions in a very specific order.

Let's walk through that interrogation. Question one. Who owns this? If I'm the auditor and I walk into Pauline's office, my first move isn't to look at her data.

I'm going to ask her who is actually on the hook for this process. And if she points to a department or says the AI committee, she fails before I even open a spreadsheet. Correct.

Question two. When did this last run and can you show me the output? This is where controls that exist only in PowerPoint presentations are vaporized. Because the evidence artifact either has a real recent cryptographic timestamp on it, or it doesn't.

You cannot fake a missing artifact while sitting in the room with the examiner. Question three. Show me the last time this failed and walk me through what happened next.

This goes directly back to the unbroken record. They want to see that your testing is adversarial enough to actually catch a failure. And they want to see the remediation plan.

They want to see the diagnosis and the treatment. And then comes the final sharpest question. Question four.

Run test again right now while I watch on a sample that I choose. This is the kill shot for a fake control. It completely removes your ability to cherry pick your best evidence or paint the bullseye.

If your control was written as a vague policy statement, it collapses the second someone outside your organization tries to run it. Because there's no real mechanism. Right.

But if you built your control using the five element framework with a specific sample size and a preset pass condition, you survived this attack. Because your test procedure was explicitly written to be executable by a stranger. That is the ultimate test of a control.

Can a stranger execute it and arrive at the exact same verdict you would? If yes, you have a control. If no, you have a vulnerability. This has been a dense, grueling journey through the mechanics of risk.

Let's synthesize exactly what we've unpacked today. We started with evolved technologies and the FTC, proving that marketing claims without verification mechanisms are not just useless, they are legally disastrous. They absolutely are.

We established our core thesis. A control must be able to fail. We detailed the five mandatory elements.

Yeah. Owner, trigger, frequency, evidence artifact, and a test procedure. We emphasized that the test procedure absolutely demands a stated sample size and a preset pass condition.

We learned that policy statements and good intentions are dangerous disguises that mask real vulnerabilities. And we covered the necessity of matching your frequency and sample size to reality, avoiding the trap of retrofitting the narrative and ensuring independent testing. It is a rigorous, unforgiving framework, but it is the only way to build governance that actually works when the pressure is on.

So as we wrap up this executive session, we want to give you, the listener, your Monday morning move. We want to give you the single most valuable concrete action you can take this week to apply what you've learned. Here is exactly what you need to do on Monday morning.

Go to your organization's compliance portal or open your AI governance file. Pick one single sentence that looks like a completed control. Don't pick a dozen.

Just pick one. Just one. Print it out.

Hand that sentence to a colleague who is not intimately familiar with this system and ask them this exact question. What exactly would have to happen for this to fail? And who would notice? What exactly would have to happen for this to fail? And who would notice? Watch their reaction. If your colleague hesitates, if they have to guess, if they start using words like probably or I assume, you do not have a control.

You have a sentence. Your immediate job is to rewrite that sentence using the five elements we discussed today before an external auditor sits across your desk and asks you that exact same question. It's about confronting the reality of your systems before reality confronts you.

Which leaves us with one final thought to mull over as you head back to your teams. We've talked a lot about the illusion of safety today. Think about it like a smoke detector in your house.

Okay, a smoke detector. Having absolutely no controls in your organization is like not having a smoke detector at all. You know you're unprotected.

It's incredibly dangerous, but at least your anxiety matches your reality. You are aware of the risk. But having untestable controls.

Having policy statements masquerading as controls. That is like having a smoke detector bolted to your ceiling that has no test button and no battery indicator. You stare up at it and you feel totally safe.

You sleep soundly right up until the moment the house is on fire. Smoke fills the room and the alarm remains completely silent. You only discover the mechanism is broken when it is too late to survive the failure.

Exactly. Don't let your governance file be a ceiling full of broken smoke detectors. Demand the test button.

Engineer the evidence. Ensure your controls can actually save you when the fire starts.