Attacking to learn: you red-team a governance file and discover how thin most are
The short answer
Execute the file, do not debate it
The strongest attack takes a claim at its word, runs the claim's own logic on the claim's own kind of data, and shows what it actually does. Lum and Isaac did not argue that PredPol was biased; they ran it and it produced a feedback loop. Behavior beats rhetoric, and a file is strong on rhetoric and thin on behavior.
What you will be able to do
- Decompose any governance file into its three structural parts (the claims it makes, the evidence it offers, and the assumptions it never states) so that the load-bearing joint becomes visible.
- Distinguish a claim that is supported by diagnostic evidence from a claim that is merely asserted, restated, or backed by evidence that would look identical whether the claim were true or false.
- Apply the re-run move (execute the file's own logic on its own data and observe the result) to a real governance claim, following the method Lum and Isaac used against PredPol in 2016.
- Detect a feedback loop in a governed system: a place where the system's own output becomes a future input, so that the evidence for the system is manufactured by the system.
- Expose a decorative control: a gate cited in the file as evidence that has never once blocked or changed an outcome, using the four questions that separate a control which governs from one which decorates.
- Rank the weaknesses you find by how much weight they carry, and name the single attack that, if it lands, collapses the file's central promise.
- Produce a red-team teardown of an external governance file: a short, evidence-anchored memo that states each claim, the attack against it, and the one weakness that matters most.
- Rank a set of findings by weight rather than count, so your teardown leads with the one weakness that collapses the file's central promise and cuts the cosmetic ones.
- Recognize what a governance file that survives the attack looks like, using the six marks of a strong file, so you know the standard your own file must meet.
- Judge the ethics of the attack: steelman before you strike, attack the file and not the author, and handle a live system responsibly, including telling a genuinely diagnostic test from one a vendor built to pass.
The ideas, one by one
Most governance files are thin, and predictably so
Across domains and jurisdictions, files fail at the same six joints: assertion in place of evidence, non-diagnostic evidence, circular evidence, unstated assumptions carried as facts, scope laundering, and a missing adversarial section. Knowing the joints tells you where to push before you have read the specific file.
Read the file as a structure, not a story
Take it apart into claims, evidence, and assumptions. The load-bearing joint, the one claim everything depends on resting on one weak assumption, becomes visible the moment the file stops being prose and becomes three columns. Break that joint, not the margins.
The counterfactual test is the fastest attack you have
For any claim and its evidence, ask what the evidence would look like if the claim were false. If the answer is "the same," the evidence is non-diagnostic and the claim is unsupported no matter how much data sits behind it. You can run this test on paper, in a meeting, in seconds.
A feedback loop turns a file's strength into its collapse
When a system produces the data used to validate it, the validating evidence agrees with the system because the system caused it. The mountain of confirming data becomes your finding rather than the file's defense. Trace one decision from output to input and see whether the loop closes.
Fairness makes the attack stronger, not softer
Steelman the claim before you strike, attack the file and not the author, and handle live systems responsibly. These are not concessions; they remove every avenue by which a defender could dismiss your finding as unfair, and they are what let your teardown survive the question "were you being fair?"
You attack others' files to harden your own
The purpose of this topic is not to become a critic. Every attack you catalog against someone else's file is an attack your own conformity file must now survive. The catalog you build here is the input to the rebuild in Topic 11.6, so the next red team that comes for your file finds less. (see Topic 11.6)
End the attack with the cheap test
A strong teardown does not stop at rejection; it names the specific, low-cost, diagnostic test that would settle whether the claim holds. This turns an attack into a path forward and demonstrates that you were attacking to learn, not to win.
Hunt decorative controls, not only missing ones
A control that has never once blocked anything deserves scrutiny. Ask of every gate the file cites: how many times has it been exercised, how many times did it change an outcome, what happened the one time it did, and if it never fired, is that because the risk never arose or because the threshold sits where nothing can reach it. A control that is present, cited as proof, and inert is thinner than an obvious hole, because the hole is visible and the decoration is not.
"Has a launch ever been delayed on a governance finding?" is the fastest read on whether governance is real
It is one question, it needs no data access, and the answer is legible: an organization that can name the launch it stopped has governance that changes outcomes, and one that cannot, while holding a thick file, has a file rather than a function. Ask it of the file you are attacking, then ask it of your own organization.
Weight beats count
One finding that collapses the central promise is worth more than twenty findings that do not, because a long list invites triage while a broken load-bearing joint forces a decision. Rank every weakness by whether the file's promise survives its failure, lead with the heaviest, and cut the cosmetic ones.
Most of the attack needs no system access
Four of the six weak joints (assertion, non-diagnostic evidence, scope laundering, and the missing adversarial section) are visible from the file alone. The counterfactual test runs on a printed page in a meeting. You can make a decisive finding before anyone gives you data or code.
Listen to it
The same lesson, talked through by two hosts as episode 86 of the podcast.
Real cases
These examples show the attacker's read applied to real governed systems. The PredPol case is treated in depth because it is the anchor for this topic; the others show that the same weak joints recur across domains and jurisdictions. In each, the point is the method, not the verdict.
Example 1: PredPol and the re-run that broke it (Lum and Isaac, Significance, 2016). PredPol grew from a Los Angeles Police Department and University of California, Los Angeles project led by Professor Jeff Brantingham, running a patented algorithm adapted from the epidemic-type aftershock sequence model used to predict earthquake aftershocks. Its governance story rested on a single load-bearing claim: because the model uses only locations and times of reported crime, it cannot encode racial bias. Kristian Lum and William Isaac took that claim at its word and ran the model's logic on Oakland drug-crime data. They brought in one outside measurement the model never sees: the 2011 National Survey on Drug Use and Health, which indicated illicit drug use was spread fairly evenly across Oakland, while recorded drug arrests were heavily concentrated in a small number of poor and minority neighborhoods. Running the mechanism forward produced a feedback loop: predictions sent police to already over-policed neighborhoods, the resulting arrests raised the recorded crime there, and the next prediction sent them back. In their simulation, Black residents were targeted at roughly twice the rate of white residents, and other non-white residents at about one and a half times the rate, from a model that claimed never to see race. The attack broke every one of the file's defenses at once: it granted the honest best case (steelman), it ran the model's own logic on the model's own kind of data (re-run), and it named the loop where arrests, the model's output, became recorded crime, the model's input (follow the loop). PredPol was later renamed Geolitica in March 2021, shortly after the LAPD discontinued its use in April 2020; subsequent independent analysis of the tool's predictions (reported by The Markup and Gizmodo in 2021) continued to find the disparity the 2016 paper had predicted on paper years earlier. The paper is the model for this entire topic: it did not debate the file, it executed it.
Example 2: A hospital vendor's "critical hallucination rate." When a healthcare AI vendor represents a specific, reassuring error metric for a clinical tool, the attacker's first move is the scope-and-diagnosticity test. What population was the metric measured on, and is it the population where the tool is deployed? Would the number look different if the tool were in fact dangerous in the field? A metric measured on curated cases, reported by the party that profits from a low number, invites the counterfactual test from Section 3D: if the true field behavior were bad, would this particular measurement have shown it? The deep treatment of the eval-suite discipline that catches exactly this, and the enforcement action that followed one such misrepresentation, belongs to Topic 4.2; here it is an illustration of Joint 2 and Joint 5. (see Topic 4.2)
Example 3: Welfare and benefits risk-scoring files (multiple jurisdictions). Governance files for automated fraud-risk and benefits systems in several countries share a recurring load-bearing assumption: that the historical records used to train the risk score reflect actual fraud rather than actual investigation. If a group was investigated more in the past, it generated more recorded findings, and a model trained on those findings will score that group as higher risk, sending more investigation its way, which generates more findings. It is the PredPol loop in a different domain. The attacker does not need the file's internal numbers to make the finding; the loop is visible from the structure alone. The deep, sourced treatment of specific welfare-scoring cases and the fundamental-rights assessment they require is owned by Module 10. (see Topic 10.4) The transferable lesson here is that Joint 3 recurs wherever a system's enforcement generates the data that trains the next round of enforcement.
Example 4: A resume-screening tool's "we removed the biased feature" file. A common governance claim states that a hiring model was made fair by deleting protected attributes such as gender from its inputs. The attacker's read finds the unstated assumption (Joint 4): that the remaining features carry no information about the protected attribute. In practice, other features (schools, hobbies, gaps in employment, even patterns of word choice) can act as proxies, so the model reconstructs the protected attribute it was never given and discriminates through the proxy. The counterfactual test is quick: if the model were still biased through proxies, would "we removed the gender field" detect it? No. So the evidence is non-diagnostic for the claim. The deep case and enforcement history of hiring-AI bias is owned elsewhere in the program; here it is an example of how deleting an input is asserted as a fix and does not survive the assumption pile.
Example 5: A frontier vendor's model card as a governance file. Not every governance file is thin, and reading a strong one teaches as much as breaking a weak one. A serious model or system card names the evaluations that were run, the populations they covered, the failure modes the authors found, and the conditions under which the model should not be used. When you apply the attacker's read to a card like that, you find that the claims have diagnostic evidence attached, the scope of testing is drawn honestly around the deployed system, and there is an adversarial section describing how the authors made the model misbehave. That is what a file that survives looks like: it red-teamed itself before you arrived. Reading it, you learn the standard your own file in Topic 11.6 has to meet. (see Topic 11.6) The discipline of reading the card itself, rather than a press summary of it, is Topic 12.1's owned scope. (see Topic 12.1)
Example 6: The absent adversarial section as the whole finding. Across many real files, the strongest finding is not a broken claim but a missing one. A file that describes a deployed AI system and contains no section on how it fails, no red-team result, no "here is what we tried that broke it," has told you that the adversarial work was not done or not disclosed. You do not need to run anything to make this finding; the table of contents makes it for you. This is Joint 6, and in the base-rate sense it is the most common serious weakness in real governance files, because writing down how your own system fails is the part everyone is tempted to skip.
Example 7: The pilot that laundered its scope. A recurring pattern in vendor files is a glowing result from a controlled pilot, presented as evidence for a system that will run at scale in messier conditions. The pilot ran in one site, with attentive staff who knew they were being watched, on a curated population, for a short window. The deployed system runs across many sites, with ordinary staff, on the full population, indefinitely. The attacker draws one line around exactly what the pilot measured and a second around exactly what will ship, and the two lines are different shapes. Every number inside the pilot line is real; none of it supports a claim about the deployment, because the tested thing and the shipped thing are not the same thing. This is Joint 5, scope laundering, and it is easy to miss because the evidence is genuine, recent, and quantitative; only the substitution is silent.
Example 8: The benchmark that tested a different model. A vendor's file supports a safety claim about the shipped product with strong scores from a named public benchmark. The attacker checks one thing before reading the scores: was the benchmark run on the exact model that ships, in the exact configuration customers get? Often it was run on a larger, more expensive internal version, or an earlier checkpoint, or with settings no customer will use. The scores are real and the model that earned them is not the model on sale. This is Joint 5 again, and it recurs so often with AI benchmarks that "which model, which configuration, run by whom" is the reflexive first question, before any number is discussed. The deeper discipline of reproducing a vendor's benchmark yourself, and a case where a benchmark's funding was undisclosed, is owned by Module 12. (see Topic 12.1)
Example 9: The endorsement page as non-evidence. Many files include a page of endorsements: named experts, advisory board members, or a certification logo. The attacker treats the endorsement page with the counterfactual test like any other evidence. If the system were unsafe, would these endorsers have known, and would they have withdrawn their names? Usually the endorsers reviewed an early version, or lent their name to the organization rather than the specific system, or hold no ongoing responsibility for the system's behavior. An endorsement that would remain on the page whether or not the system is safe is non-diagnostic. This does not mean endorsers are dishonest; it means a name is not a measurement, and a governance decision cannot rest on one.
Example 10: The certification that certified the process, not the system. A file cites a management-system certification (of the kind that attests an organization runs a defined process for governing AI) as evidence that a specific model is safe or fair. The attacker separates the two claims. A process certification attests that the organization has procedures, documents decisions, and reviews risks; it does not measure whether any particular system behaves well, and it would remain valid whether or not the specific model discriminates. Using a process certification to support a system-behavior claim is a scope swap (Joint 5) dressed in an authoritative logo. The certification is real and valuable for what it attests; it is non-diagnostic for the behavior claim it is being borrowed to support. The deep treatment of what such management-system standards do and do not certify is owned by Module 6; here it is an example of authority standing in for evidence.
Example 11: The oversight that only looks like oversight. Many files claim a human is in the loop: "every automated decision is reviewed by a staff member before it takes effect." The attacker does not dispute that a human is present; the attacker asks what the human actually does. If the reviewer sees hundreds of decisions an hour, has no independent information beyond what the model shows, faces no consequence for agreeing and friction for disagreeing, and approves at a rate near 100 percent, then the oversight is nominal: the human is a rubber stamp that launders the model's decision into a "human" one. The counterfactual test applies directly: if the model were making harmful decisions, would this oversight process catch them? When the honest answer is no, the "human in the loop" claim is non-diagnostic, and the finding is that the file describes the presence of a human, not the exercise of oversight. The deep treatment of where a human must genuinely sign, and the enforcement action behind it, is owned elsewhere in the program; here it is an example of a claim that reads as strong and tests as hollow.
Where people go wrong
- "A strong-looking file is a strong file." The opposite is often true. The files with the most confident language and the most impressive numbers are frequently the thinnest, because polish is cheap and diagnostic evidence is expensive. A glossy file with a mountain of confirming data may be resting entirely on a feedback loop (Joint 3), and the mountain is the finding, not the defense.
- "To attack a file I need its internal data or code." Many of the strongest findings need nothing but the file's own structure. The missing adversarial section (Joint 6), the scope swap between demo and deployment (Joint 5), and the unstated assumption carrying the whole claim (Joint 4) are all visible from the outside. Lum and Isaac needed public arrest data and a public survey, not PredPol's source code, to break its central claim.
- "Attacking the file means proving the authors lied." A strong attack works even if the authors were completely honest. Most thin files are written sincerely by capable people who made an invisible assumption. An attack that requires bad faith is weak, because the authors can simply and truthfully deny bad faith. Grant them their best case and break the file anyway.
- "Accuracy is evidence of fairness." Accuracy measured against a system's own outputs is evidence of a loop, not of fairness or even of accuracy in the sense that matters. When the system chooses which cases get recorded, high agreement between predictions and records is guaranteed and meaningless. Always ask what generated the ground truth the accuracy is measured against.
- "If I cannot run the system, I cannot use the re-run move." The re-run has a paper cousin that needs no execution: the counterfactual test. For any claim and its evidence, ask what the evidence would look like if the claim were false. If it would look identical, the evidence is non-diagnostic and the claim is unsupported. You can run this test on a printed file in a meeting.
- "Finding many small weaknesses is a strong attack." A long list of minor flaws is easy to dismiss and easy to fix around. A strong attack finds the one load-bearing joint whose failure collapses the central promise, and spends its force there. Rank your findings by how much weight each carries, and lead with the one that matters.
- "Removing a sensitive input makes a model fair." This is the proxy misconception. A model denied a protected attribute can reconstruct it from correlated features and discriminate through the proxy. "We deleted the gender field" is non-diagnostic for the claim "the model does not discriminate on gender," because the model can discriminate through everything else it still sees.
- "An academic paper-only attack does not apply to a real deployed system." The PredPol case is the direct refutation. Lum and Isaac's 2016 paper-only re-analysis predicted the exact disparity that independent investigations of the live, deployed tool documented years later. A rigorous attack on the logic of a system is an attack on the system, because the deployed system runs that logic.
- "Being fair to the file makes the attack softer." Steelmanning, attacking the file rather than the author, and handling live systems responsibly make the attack stronger, not softer, because they remove every avenue by which the attack could be dismissed as unfair. Fairness is not a concession. It is what makes the finding stick.
- "The goal of the teardown is to reject the system." The goal is to learn where files are thin and to make systems, including your own, stronger. A good teardown often ends with the cheap, specific test that would settle the question (as Gretchen's did), not with a verdict. You are attacking to learn, and the best outcome is a better file, sometimes theirs and always yours.
- "A human in the loop means real oversight." The presence of a human is not the exercise of oversight. A reviewer processing hundreds of decisions an hour, with no independent information and every incentive to agree, is a rubber stamp. Ask what the human actually does and whether the process would catch a harmful decision; if it would not, the oversight claim is non-diagnostic no matter how many humans are named.
- "The controls the file lists are the part I can take at face value." They are the part most worth attacking. Hunt decorative controls, not only missing ones: a gate that exists on paper, is cited as evidence, and has never once blocked or changed anything is an assertion whose evidence turns out to be another assertion. Ask how many times it has been exercised, how many times it changed an outcome, what happened the one time it did, and if it never fired, whether the risk never arose or the threshold sits where nothing can reach it. A perfect pass rate over years is a finding, not a reassurance.
- "A missing control is a worse finding than a weak one." Usually the reverse. A missing control is visible to everyone, including the file's authors, and it gets fixed. A decorative control is invisible precisely because it is present, and it buys the organization the appearance of governance at no cost in outcomes. The related organizational tell is the fastest test you have: ask whether a launch here has ever actually been delayed or stopped on a governance finding, and ask them to name it. An organization that can name one has real governance; one that cannot, with a thick file, has decoration.
- "I should attack the file in the order it is written." Reading front to back keeps you reacting to prose, which is where the file is strongest. Sort it into claims, evidence, and assumptions first, find the load-bearing joint, and start there. The order of your attack is set by weight, not by the file's table of contents.
- "A confident tone signals a strong file." Tone is free and evidence is expensive, so confidence and thinness often travel together. Bold, reassuring language is a reason to look harder at the evidence beneath it, not a reason to relax. The files that most need attacking are frequently the ones that sound most certain.
- "The deployment population is different, so the training-data results do not apply, and that is a fair excuse." This is scope laundering wearing a disguise. A file's central claim is about the deployed system; if the vendor's own defense is that the deployment population differs from the population the evidence covers, the vendor has just conceded that the evidence does not test the claim. The correct response is not to accept the excuse but to name it as Joint 5 and require evidence measured on the population actually being deployed to.
- "If the vendor refuses my cheap test, there is nothing more I can do." A refusal to run a cheap, fair, diagnostic test is itself a finding, not a dead end. Write the refusal into the teardown explicitly, and if the stakes justify it, escalate past the vendor relationship: request an independent third-party audit, invoke a right-to-audit or evaluation clause in the procurement contract, or route the finding to the relevant oversight body. A vendor who will not let their central claim be tested has told you something a passed test never could.
Questions people ask
- What is governance file?
- The assembled documentation that is supposed to show a system is fair, accurate, safe, and overseen: claims about the system plus the evidence and records offered to support them. In this program the conformity file assembled in Topic 5.6 is the canonical example. A file is strong when its claims survive being tested and thin when they collapse under an attacker's read.
- What is red-team teardown?
- The artifact this topic produces: a short, evidence-anchored memo that decomposes an external governance file into claims, evidence, and assumptions, names the single load-bearing weakness, states the one attack that breaks it, and ends with the cheap diagnostic test that would settle the question.
- What is the attacker's read?
- The method of reading a governance file as a structure rather than a story, by sorting it into three piles (claims, evidence, assumptions) and locating the load-bearing joint, instead of reacting to the file's prose.
- What is load-bearing joint?
- The single claim, or the single assumption beneath it, that everything else in the file depends on. Breaking the load-bearing joint collapses the file's central promise regardless of how sound the remaining claims are. The attacker's goal is to find and break this joint, not to list every minor flaw.
- What is claim?
- A load-bearing promise a file makes about a system, stated as a plain sentence that could be true or false, with the adjectives and reassurance stripped away. Claims can be tested; adjectives cannot. More on Claim
Keep going
This lesson builds How models work, at a governance depth, and that page shows the roles that hire for it. Every Certified AI Governance Professional (CAIGP) lesson.
Watch the lesson and prove it
Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.
Watch the lesson and prove it →