Attacking to learn: you red-team a governance file and discover how thin most are

Imagine you are sitting in a boardroom, right, and an organization has just spent millions of dollars, maybe several years, building this advanced automated system. Yeah, a massive capital investment. Exactly.

And the explicit mandate from day one, the thing everyone agreed on, is that this system must be entirely neutral. Right, they want zero bias. Zero bias.

So the engineers strip out every demographic variable, they ban the model from ever looking at race, gender, income, all of it, and the vendor hands you this dense, perfectly formatted governance file. I know the exact kind of file you mean. High gloss, very professional.

Oh, absolutely. And it claims it is mathematically impossible for the system to be biased. So the executives sign off, the system goes live, and immediately, like day one, it begins targeting minority populations at twice the rate of anyone else.

It sounds like this crazy, catastrophic edge case, right? But in the landscape of algorithmic governance, it's, well, it is actually the norm. Which is terrifying. It is.

What you're describing is this fundamental disconnect between a system's rhetorical defense, you know, the story it tells about itself, and its actual operational reality. And that's exactly what we are getting into on today's Deep Dive. We are looking at this purely from the perspective of the reviewer.

Right, the person sitting across that mahogany table. Yes, you are the busy professional. A vendor has just handed you that 50-page, high-gloss document that confidently assures you their new AI model is accurate, it's safe, and it's fair.

But your job isn't to just read that document and nod. No, your job is no longer to read that document. Your job is to break it.

I love that framing. Because today's focus is on adversarial governance. Specifically, how to red-team a governance file, right? Yeah, to discover its critical, load-bearing weaknesses before that piece of technology is ever deployed.

And to do this, we really have to recognize that the skills required to defend a system are entirely different from the skills required to attack one on paper. Completely different skill set, yeah. You aren't being a cynic just to be annoying.

You're learning the anatomy of a strong file by breaking a thin one on paper before it breaks in the real world and hurts people. Right. We're going to treat this as an executive-level teardown.

We are looking at the exact frameworks used to dismantle these confident vendor claims. And we have to start with the baseline principle of this entire discipline. Execute the file.

Do not debate it. Exactly. Execute the file.

Do not debate it. That principle, it's the dividing line between amateur reviewers and professional attackers. Because amateurs argue with the adjectives, right? They read the vendor's brochure and they argue about the wording.

Right. They get caught up in the philosophy of it. But professionals look at the underlying mechanics.

And to see what this looks like in practice, we have to look at this absolute watershed moment in algorithmic auditing. This is the 2016 paper, right? Yeah. Published by researchers Christian Loom and William Isaac.

It was in Significance, which is the magazine of the Royal Statistical Society. Great title too. To Predict and Serve.

It's an incredible title. And the target of that paper was a widely deployed predictive policing product called PredPol. Which, I mean, the origins of PredPol are wild.

It's deeply rooted in academia and municipal budgets. Yeah. It emerged from a collaboration between the Los Angeles Police Department and some researchers at UCLA, led by Professor Jeff Brantingham.

And they took the underlying mathematics from a completely different field, right? They completely repurposed it. They did. They took an algorithm originally designed to model HAWQS processes.

Right. Which seismologists used to predict earthquake aftershocks. And they applied it to crime.

Which, I mean, the theory makes a certain kind of intuitive sense if you are looking at very specific types of property crime. Like burglaries. Right.

In seismology, an earthquake relieves tectonic stress, but it transfers that stress to the surrounding area, which causes aftershocks. So the UCLA team theorized that crimes like burglary work similarly. Exactly.

A broken window on a street signals a vulnerability. That invites more burglaries in that immediate vicinity over the next few days. So the model learns those spatial and temporal clusters.

But the vendor didn't just sell it for burglaries, did they? No. They expanded its scope massively. Yeah.

And they wrapped it in this incredibly robust governance story. The core claim of the PredPol file was that the model was entirely objective. They literally said, math does not have a race.

Math does not have a race. That was the claim. Their primary defense was that the algorithm only ingests three data points, right? The type of crime, the location, and the time.

So the vendor stated explicitly that because the model never sees demographic data, it cannot be biased. It is mathematically impossible. But Lum and Isaac didn't write an op-ed debating the philosophy of blind algorithms.

They didn't argue with the brochure. No. They did something much more dangerous to the vendor.

They took the governance claim entirely at its word and they executed it. The rerun move. Yes.

The rerun move. They fed the PredPol model real drug crime data from Oakland, California. But the critical methodological choice they made, and this is so important, they introduced an external independent ground truth.

Right. To compare against the police department's internal data. Exactly.

They used the 2011 National Survey on Drug Use and Health. The NSDUH. Yeah.

It's a massive, rigorous public health survey. And the data from that survey revealed a reality that literally anyone working in public health already knew, which is that illicit drug use is ubiquitous. Yeah.

It's everywhere. It was spread almost perfectly evenly across the entire geography of Oakland, across all demographics. But when Lum and Isaac plotted the recorded drug arrests made by the Oakland police, the map looked entirely different.

Completely different. The arrests were hyper-concentrated in a small number of lower-income, predominantly minority neighborhoods. And this gets to the core issue.

This is the difference between victim-reported crime and police-discovered crime. Right. Because if your house is burglarized, you call the police, regardless of what neighborhood you live in.

Exactly. So, the police data for burglaries closely mirrors the actual occurrence of the crime. But drug possession is, well, it's a victimless crime in the reporting sense.

No one calls the police to report their own drug use. Obviously not. So, a drug crime is only recorded if a police officer happens to be standing there to witness it.

The arrest data doesn't map where people are using drugs. It maps where police officers are geographically deployed. Bingo.

It maps the historical patrol patterns. So when Lum and Isaac ran the earthquake algorithm on this specific dataset, the mathematical results were severe. The model looked at the historical arrest clusters and predicted that future drug crimes would happen in those exact same neighborhoods.

Which means it sent the police right back there. Yeah. In the simulation, the system directed police to target black residents at roughly twice the rate of white residents and other non-white residents at about 1.5 times the rate of white residents.

So the vendors claimed that the math didn't see race. It was technically true, right? But operationally meaningless. Exactly.

By relying on police-discovered crime data, the system was simply laundering historical deployment patterns through this complex mathematical earthquake algorithm. The model was learning the behavior of the police department, not the behavior of the community. It is the equivalent of a chef presenting you with this beautifully written 20-page menu detailing their complex culinary philosophy.

Right. All the adjectives. All the adjectives.

And instead of debating their choice of words, you simply ask them to sit down and actually eat their own food. Just eat the recipe. I love that analogy.

It shifts the fight from the ground of rhetoric, where the vendor's file is strong, to the ground of behavior, where the file is incredibly thin. And when Lum and Isaac did this, they exposed a terrifying reality for anyone sitting on a governance review board. Because the PredPol case wasn't just a one-off bad apple.

No. Not at all. It reveals a terrifying base-rate truth.

Most governance files, even from serious organizations with real lawyers, are fundamentally thin, and they fail in the exact same ways. Yes. When we say a file is thin, we are talking about a very specific phenomenon where a document reads as highly comprehensive, authoritative, complete.

It looks great. It looks amazing. Yeah.

But its structural integrity just collapses the moment it is subjected to mechanical testing. And this thinness is not random. It gathers reliably at six specific weak joints.

Let's go through those six joints, because this is where the actual attack happens. Right. So joint number one is the assertion in place of evidence.

This happens when a governance file states a design intention, as if the intention itself were a proven operational result. So you'll see phrases like, the system is built to avoid bias, or the model is designed to prioritize human safety. Exactly.

Now, in traditional deterministic software engineering, if you write code to execute a specific function, the intention usually matches the output. It does what you coded it to do. But in probabilistic systems, like machine learning, intention is entirely decoupled from the system's eventual behavior in the wild.

Right. An attacker reading that file recognizes that an intention is merely a starting condition. If the vendor cannot produce a specific empirical measurement, like a log, a behavioral test, an audit trail, that proves the intention survived the training process and actually exists in the deployed model.

Then the claim is treated as an empty assertion. The document is essentially offering the blueprint as proof that the building didn't collapse. Okay, joint number two is non-diagnostic evidence.

And we should define what diagnostic evidence actually is, right? Yeah, diagnostic evidence behaves like a functioning medical test. It yields one specific measurable result if the system is healthy, and a distinctly different measurable result if the system is failing. It has the capacity to fail.

Right. Non-diagnostic evidence, by contrast, looks absolutely identical, regardless of whether the system is functioning perfectly or causing massive harm. And the most common way we see this is the misuse of accuracy metrics.

Oh, all the time. A vendor will proudly display a 95% accuracy rate. But if you look at the methodology, that 95% was achieved by testing the model on a holdout set of the exact same data distribution it was trained on.

They're testing it in a sterile laboratory environment. Right. But the claim they are trying to support is that the model will be safe and accurate when deployed in the chaotic, shifting environment of the real world.

So that 95% lab score is real data, but it is non-diagnostic for the claim being made. Because the model could be failing catastrophically in deployment, and that historical lab metric would still read 95%. It doesn't change.

Exactly. It's like pointing to a thermometer that has been painted to always read 72 degrees and using it to claim the house isn't freezing. That brings us to the third joint, which is circular evidence.

We're going to dive really deep into this one in Section 5, but just broadly. What is it? Broadly, circular evidence occurs when a system is validated using outputs that the system itself generated. It's a feedback loop, and it's deadly because it looks like quantitative strength.

Okay, holding on to that one. The fourth joint is unstated assumptions carried as facts. Right.

Every governance file rests on a foundation of load-bearing premises that the authors rarely even realize they are making. Because data scientists and engineers are trained to treat data as ground truth, they often fail to recognize that data is merely a low-resolution shadow of human behavior. Perfect way to put it.

In the Pritiple case, the unstated assumption was reported drug arrests represent actual drug usage rates. And in algorithmic hiring tools, we see this all the time. The unstated assumption is often historical promotion rates are a pure reflection of employee merit.

Which completely ignores historical systemic biases against women or minorities in that company. The reviewer's job is to extract these invisible assumptions, put them under bright lights, and ask, do these actually map to physical reality? Because if the assumption fails, the entire mathematical structure built on top of it collapses. Absolutely.

Moving to the fifth joint, scope laundering. Scope laundering. I love this term.

What does it mean? It's a highly specific maneuver where the vendor swaps the parameters of the system that was tested for the parameters of the system that is actually being sold. You see a lot of this in medical AI, right? A vendor builds a diagnostic algorithm to detect, say, early onset sepsis. They test it in a heavily funded Tier 1 research hospital utilizing perfectly calibrated, state-of-the-art monitoring equipment.

With nurses who are explicitly trained to format the data perfectly for the AI? Right. And in that specific narrow scope, the tool achieves a 99% success rate. But then the vendor takes that 99% accuracy claim and uses it to sell the software to an underfunded rural clinic network.

Where the environment's completely different? Completely. You've got 15-year-old hardware, incompatible data standards, and overworked staff that cannot meticulously format inputs. The vendor is laundering the success metric from the highly controlled pilot scope into the chaotic deployment scope.

So the test for this is simple. You draw a hard mental boundary around the exact conditions of the test. Then you draw a boundary around the exact conditions of the proposed deployment.

And if those two shapes are not perfectly identical, the evidence is being laundered. Which brings us to the sixth and final joint, which might be the most obvious when you know to look for it. The missing adversarial section.

Right. If you have a 50-page governance document and it contains literally no section detailing how the system fails, which edge cases break it, or what stress tests the engineering team attempted that resulted in poor outcomes, that absence is your primary finding. Because complex automated systems do not operate flawlessly across all parameters.

It's impossible. Exactly. If the failure modes are absent from the file, it guarantees one of two realities.

Either the vendor lacked the competence to look for them, or they found them and made a conscious decision to hide them from the review board. Let me push back on this a bit, though, because as a reviewer, you get handed this glossy, beautifully formatted file. Isn't a highly polished, confident tone usually a sign that an organization actually has its act together? Like they spent the money on good tech writers and compliance people.

I get that instinct. But there is a psychological trap here for the reviewer. Human nature equates that polish with engineering competence.

But in adversarial governance, you must recognize that tone is free. Tone is free. Right.

Confident prose costs nothing. Diagnostic adversarial testing, on the other hand, is incredibly expensive. It's time consuming, and it often yields really uncomfortable results for the vendor.

Therefore, confidence and thinness often travel together. A hyper-confident tone is frequently a camouflage mechanism for a thin file. So if these files are built to look strong through confident prose, how do you, as a reviewer, physically read them so you don't get fooled? You have to stop reading them like a book.

Because if you read it front to back, you're just absorbing a story. Exactly. Corporate prose is explicitly designed to soothe the reader.

It maintains a smooth narrative flow, guiding your brain past logical gaps, using industry jargon and reassurances. Amateurs read front to back and react to the prose. Attackers read the file as a structure.

So how do we physically do that? You dismantle the file into three distinct columns. Just strip away the narrative entirely. Okay, so what's the first column? Pile one is the claims.

You comb through the document and extract only the load-bearing promises. You remove every adjective, every marketing buzzword, every statement of corporate philosophy. You reduce the text to bare, testable propositions.

Things like, the model does not utilize protected attributes. Or a human operator reviews every flagged decision before execution. Right.

Or the system processes queries with a 99% uptime. You are isolating the bare metal of the contract. Okay, what's pile two? Pile two is the evidence.

For every single bare claim you extracted in pile one, you search the document for the specific empirical evidence offered to support it. So you have to ruthlessly separate actual system logs from just future promises. Exactly.

Sort out the design descriptions from actual rigorous statistical measurements. And what you're really looking for here are the empty spaces. The moments where a massive load-bearing claim in pile one has absolutely nothing mapped to it in pile two.

Just a completely unsupported promise. Yeah. And then you build pile three, which is the assumptions.

This requires the most analytical rigor. Because the assumptions aren't usually written down, right? Exactly. For every paired claim and piece of evidence, you ask a singular question.

What specific conditions must be true about the real world for this piece of evidence to legitimately validate this claim? And you write these down as testable claims. So it's basically reverse engineering a magic trick, isn't it? Oh, that's exactly what it is. You stop looking at the magician's hands, the beautiful prose, the formatting, and you just look for the trapdoor.

And the trapdoor is usually in that assumptions pile because nobody bothered to defend it. Right. The vendor almost never defends the assumptions pile.

They usually don't even know it exists. So once you have the file sorted into this structure, these three piles, your objective becomes highly targeted. You aren't trying to find 20 minor typos.

You are hunting for the load-bearing joint. The single claim or the one weak assumption beneath it that carries the entire central promise. Because you don't need to break every claim in the file.

You only need to break this joint. If you break that one joint, the entire file collapses. So once we have the file sorted and we found our target, we need a weapon to attack the evidence pile.

Yes. And as a reviewer sitting in a boardroom, you don't always have the source code or the data to execute a live rerun like Lum and Isaac did, right? Right. You rarely have access to the API or the training data in that moment.

So what's the weapon? What is the paper equivalent of the rerun that we can use right there in the meeting? The counterfactual test. It is the fastest attack you have. It forces the reviewer to invert their perspective.

Instead of asking, does this evidence prove the claim is true? You ask, if this claim were entirely false, would this evidence look any different? Let's apply this to some real world examples because I think this is where the framework really clicks. Let's talk about the internal review defense. So say a vendor is selling a human resources platform.

They claim the algorithm's sorting mechanism is entirely neutral and fair. The evidence they provide in pile 2 is a statement reading, our internal compliance team conducted a rigorous fairness review and found zero issues. Okay, so we run the counterfactual test.

We ask, if this claim were false, if the algorithm was actually heavily biased, what would an internal review run by the vendor with vendor incentives have found? It would have found the exact same clean bill of health because the internal review is structurally designed to find compliance, especially when they are trying to close an enterprise contract. So the evidence is non-diagnostic. It looks identical whether the system is fair or biased.

Exactly. Consider another highly prevalent defense, the deleted proxy. Let's stay with a resume screening tool.

The vendor claims fairness because, and this is their evidence, we explicitly deleted the gender field from the training data. The model does not know the applicant's gender. Apply the counterfactual.

If the model were still actively discriminating against women, would the fact that they deleted the gender field look any different? No, it wouldn't. Because the model can still reconstruct gender from proxies, right? Exactly. High dimensional models are pattern matching engines.

They don't need explicit labels. They look at the ZIP code, the specific university attended, the extracurricular activities. Like playing women's field hockey.

Right. Or even the semantic vocabulary choices in the cover letter. The model synthesizes these proxies to reconstruct the applicant's gender with terrifying accuracy.

So the model discriminates based on the reconstructed proxy completely bypassing the deleted field. Yep. The vendor's evidence, we deleted the field, is factually true but operationally non-diagnostic.

It proves nothing about behavior. We see this same failure mode with endorsement pages, don't we? You flip to the back of the vendor's file and it's just a page full of impressive logos. ISA certifications, university partnerships.

Right. Do logos of certifiers prove safety? Apply the test. If the specific localized deployment of this algorithm was operating unsafely and causing harm today, would those logos still be printed in the brochure? Yes, of course they would.

Because a process certification only proves an organization has procedures, not that a specific model behaves well. But let me ask you this, doesn't this mean no evidence is ever good enough? Because a vendor could say, you can always play the what-if game, you're demanding an impossible standard of perfect evidence. And that is a fundamental misunderstanding of the framework.

A fair counterfactual doesn't demand perfect evidence, it demands diagnostic evidence. Evidence that moves at least somewhat when the truth moves. Exactly.

If a vendor claims a facial recognition system is highly accurate and their evidence is an audit performed by an independent adversarial third party utilizing a demographically balanced data set in the exact lighting conditions of the proposed deployment, that is diagnostic evidence. Because if the system were biased, that specific test would return a high error rate. Right.

The test has the capacity to fail. Evidence that cannot possibly fail isn't evidence, it is marketing. Now when you run this counterfactual test, especially on accuracy numbers, you will frequently uncover the most dangerous weakness of all.

Yes, joint number three, circular evidence. A feedback loop that turns a file's strength into its collapse. Why is this so deadly? Because it looks like overwhelming quantitative strength.

The system's evidence agrees with the system perfectly. But they agree because the system produced the evidence in the first place. So how do you trace a loop? How do you actually find it in the file? You have to trace the life of one single data point.

You just ask, does an output ever become a future input? We saw this in PredPol, right? The output is where to send the police. And the police go there, they make arrests because they're present, and the next day that new arrest data is ingested as fresh training data. Output becomes input with a one day delay.

It also happens with welfare or benefits risk scoring tools. All the time. Jurisdictions train models on decades of historical fraud data to predict future fraud.

But historical fraud data doesn't represent all fraud. It only reflects historical investigation patterns. Right.

Investigated groups generate more findings, which trains the model to score them as higher risk, which generates more investigations. Which validates the model perfectly. Let's walk through an immersive scenario to really solidify this.

Let's take the fictional city of Harborview. Okay, let's do it. So you are an executive in Harborview.

You are reviewing a 40-page glossy file for a system called Resident Risk. It's an objective code enforcement tool. And the vendor claims it has a 91% accuracy rate for finding building code violations.

Okay, so as the reviewer, I'm bypassing the UI mockups. I'm looking right at that 91% accuracy claim, and I apply the loop trace. So the model analyzes historical data and outputs a high risk score for the east side neighborhood, which happens to be older and lower income.

Right, so what do these inspectors do? They follow the model. They go to the east side. And building codes are dense.

If inspectors spend a week looking anywhere, they find violations. Peeling paint. Unpermitted water heaters.

So they log all those violations on the east side, and that becomes the training data. The 91% accuracy holds solid. Meanwhile, the west side neighborhood, the affluent area, gets a low risk score, so no inspectors go there.

So the algorithm sees zero newly recorded violations on the west side, and assumes it's in perfect compliance. The 91% accuracy is just the loop congratulating itself. It is the ultimate case of grading your own homework, but worse, you're also writing the answer key while taking the test.

The mountain evidence is actually the murder weapon. I love that, yes, exactly. But when you find a loop this devastating, the temptation is to write a memo calling the vendor dishonest.

You want to expose them. And that is the worst mistake you can make. Why? Shouldn't we point out that they're lying? Because in corporate governance, fairness makes the attack stronger, not softer.

Fairness is tactical armor, not politeness. If you launch an emotional attack or you are unfair, the defender will change the subject to your unfairness. And they'll win the room.

Exactly. The executives will side with the vendor because they already spent political capital securing the budget. You have to follow the three rules of a fair attack.

What's rule number one? Steelman before you strike. You must state the absolute strongest honest version of the claim. Grant PredPol, it's best case.

Assume it is a sincere demographic free model. Because it fails anyway under the best possible assumptions, the vendor has no escape. Precisely.

The intentions become irrelevant to the mathematical reality of the failure. Rule number two, attack the file, not the author. Don't assume bad faith.

Right. Some of the thinnest files are written by capable people who just assumed the false premise was true, like reported crime equals actual crime. Give them a finding they can act on, an objective mechanical finding, not an insult they will fight.

And rule number three applies to live systems. Yes, handle live systems responsibly. If the system is live and affecting real people, you must separate the governance finding from the operational attack detail.

So the governance finding is safe to share with the board. But the operational detail, the exact sequence that triggers the harm, must be handled with extreme care and routed straight to the operator to fix. Attacking to learn must never become harming the governed population to prove a point.

Let me ask you this though. What if I do all this? I steel man their claim, I run the counterfactual, I trace the loop, and I can't break it. Does that mean I failed the red team exercise? No, absolutely not.

That means the claim is sound. And you have to endorse it. You just tell the board it passed? Yes.

A red team that invents weaknesses where none exist destroys its own credibility. If the evidence is diagnostic and the assumptions map to reality, you have a good system. But assuming you did break it, you found the structural flaw, now it's Monday morning, and you have to synthesize these findings into a document that actually forces a decision from your executive.

The Monday morning move. The biggest mistake here is giving executives a list of 20 flaws. Because a long list invites triage, right? Exactly.

They'll patch the typos, ask for a few new charts, and move on. You have to rank your findings by weight, not count. How do you do the weight test? You ask, if this were the only thing wrong, would the central promise still stand? You discard the cosmetic footnotes, and you lead with the single load-bearing weakness that collapses the promise.

You also advise reviewers to hunt for decorative controls, right? These are gates or review boards that exist on paper, but have never changed in outcome. So what's the question you ask the vendor to spot these? You just ask, has a launch here ever actually been delayed or stopped on a governance finding? If they can't name one, the governance is decorative. It's just there for show.

That's brilliant. Now, how do you end the teardown memo? You say we shouldn't just offer a flat rejection. Never end with a flat rejection.

It's too easy for an executive board to overrule. You must end constructively by proposing the cheap diagnostic test. A specific, low-cost test that would settle the truth? Exactly.

So for resident risk, the cheap test is, inspect a random sample of low-scoring blocks. Go to the Westside. And for a credit model, it would be approving a random sample of denied applicants.

Right. If they run it and pass, great, you get a good tool. But if they insist on running a rigged test-like, they choose the sample.

Or if they refuse entirely, that refusal is your final damning finding. It's the ultimate superpower for anyone sitting on a review board. From finding the load-bearing joint to running the counterfactual to proposing the cheap test.

It changes the entire balance of power in the room. So as we wrap up, what is the single most valuable move our listener should make Monday morning? Pick up one governance file, compliance document, or vendor pitch currently sitting on your desk. Ignore the prose.

Do not read the adjectives. Extract its central claim and run the counterfactual test right there on the paper. Ask yourself, if this claim were completely false, would this evidence look any different? You are attacking these files to learn.

Because tomorrow you will have to write your own governance file and now you know exactly what it has to survive. You'll know how to build a file that isn't thin. Exactly.

Thanks for joining us on the Deep Dive.