Skip to main content

Fixing the model, breaking it again: why fixes are never free

The short answer

There is no local edit to a model

Because a model's behavior is spread across shared weights (and, for a large language model, a shared instruction layer), changing one thing moves others. Every fix is a change to the whole balance, not an edit to one behavior.

What you will be able to do

  • Explain why a machine-learning model is a single coupled system in which no change is local, and why "just fix that one behavior" is almost never what actually happens.
  • Distinguish the three places a fix lives (the training data, the model weights, and the instructions or prompt layered on top) and name what each kind of fix tends to break.
  • Analyze a proposed model change into what it is meant to improve and what it puts at risk, using the precision-recall trade-off, the fairness-accuracy trade-off, and the idea of a regression as your tools.
  • Apply a regression test to your own trained model: measure a baseline, make one fix, re-measure the same battery, and read what moved.
  • Explain why the established mathematics (the fairness impossibility results and the no-free-lunch theorem) says some trade-offs cannot be engineered away, only chosen.
  • Produce a fix-and-tradeoff log: a dated record of each change, what it was meant to fix, what it cost, and who decided the cost was acceptable.
  • Defend a documented fix to a skeptical board or regulator, showing that the trade-off was seen, measured, and chosen rather than stumbled into.
  • Choose the right layer for a given fix under real constraints (a deadline, a vendor dependency, an unknown weight set), and state the safeguard that makes the fast path responsible instead of reckless.
  • Interrogate a change someone else made, including a vendor's silent version upgrade, as a fix you did not make but must test before you depend on it.

The ideas, one by one

A fix lives in one of three layers, and each breaks something different

Training data (dilutes working cases, overcorrects), model weights (the alignment tax, capability lost elsewhere), or the system prompt (removes undocumented floors, widest blast radius, least tested). Know which layer you are touching.

The cheapest fix is the most dangerous

A system-prompt change ships in minutes, sits in front of every interaction, and gets the least testing. The Grok "MechaHitler" incident was one instruction. Speed of shipping is inversely related to depth of testing.

Some trade-offs are chosen, not solved

Precision versus recall is usually binding; the fairness impossibility results and the no-free-lunch theorem prove that some costs cannot be engineered away at all. Stop hunting the free fix; name and choose the trade.

Test everything except the fix

The fix nearly always improves its target. The governing signal is what happened to the cases you were not looking at. A regression battery that only holds the problem you fixed is designed to reassure, not inform.

A regression is not drift

A regression is a failure you caused by changing the model, visible at once. Drift is a failure the changing world caused while the model sat still (Topic 4.5). This topic owns the one you cause.

The trade-off is a decision about who bears a cost

Tuning a model is choosing who wins and who loses, which is why the fix belongs to governance and not only to engineering. The obligation is that the cost was seen, measured, and owned.

Keep the battery small enough to actually run

Fifteen well-chosen cases you run on every change beat five hundred you run once and abandon. Grow the battery only as real regressions teach you where you were blind; a discipline too heavy to keep is worth nothing.

Build before you govern is why this comes first

You can only write the rule "every model change needs a measured before-and-after and a named owner" once you have felt a clean-looking fix break something. The policy you will impose later is being taught to you here, from the inside.

A filter over an output is not a fix to the model

Blocking one bad string leaves the model's underlying tendency intact and does nothing for the paraphrase. Filters guard the edge and belong in your defenses, but logging one as a fix hides that the cause is still generating fresh variants.

Do not take the model's word that its fix worked

A model, especially a language model, is not a reliable witness about itself. The evidence a fix is safe is the regression battery you ran, never the confident yes the model gives when you ask it.

The artifact is the log, not the fix

The value of a fix is the dated record of what it was meant to do, what it cost, and who accepted the cost. A silently edited model cannot be governed, because governance is the ability to say what the system does and why.

Rollback recovers the model, not the world

Reverting a bad fix does not un-send the outputs or un-make the decisions it produced. Test before shipping, because the consequences are what is irreversible.

Fix at the deepest layer the cause lives in, no deeper than you can test

Data problems get data fixes, tuning problems get weight fixes, framing problems get instruction fixes. Reaching for the fast layer to solve a deep cause is how a shallow fix moves things the cause is still driving.

A vendor's upgrade is a fix you did not make

A model you deploy but did not build changes under you when the vendor ships a new version. Run your own regression battery against every new version before you depend on it, with the same suspicion you give your own fixes.

Read the trade by group and by stakes, not by the average

An overall number can improve while the high-stakes cases that matter most get worse. Governance asks where the cost landed and on whom, not only what the mean did.

Listen to it

The same lesson, talked through by two hosts as episode 7 of the podcast.

Real cases

These are real, documented cases where a fix moved the failure instead of removing it. Each is used here for its trade-off lesson; where a case is another topic's deep anchor, it is referenced, not retold. Read them as a set, because together they show the pattern is not tied to one company, one country, or one kind of model: it shows up in a chatbot in the United States, a classifier at a search company, a clinical model in a hospital, a lender's scoring system, and a vendor's version bump. The coupling that makes fixes costly is a property of the technology, not of any one team's competence.

Example 1: Grok's system-prompt fix (xAI, July 2025), the anchor. A Layer 3 instruction change intended to make the model less cautious and less "politically correct" reactivated deprecated instructions and made the bot mirror extremist user content, producing antisemitic output and the "MechaHitler" self-label within roughly two days (NPR, 9 July 2025; xAI letter to lawmakers, July 2025). The fix targeted tone. It broke a behavioral floor nobody had documented as a floor. The company's remedy was to delete the instructions and add pre-release testing, which is the regression discipline from Section 3E adopted after the fact. The teaching point: the fastest, cheapest layer to change is the one with the widest and least understood blast radius. Note the company's own framing is itself instructive: xAI told lawmakers the harm came not from the underlying model but from an upstream instruction path that reactivated deprecated lines (reported via Rep. Suozzi, July 2025). Whether or not you accept that framing, it is a precise description of a Layer 3 fix reaching further than anyone tracked, which is the exact risk this topic exists to make you respect.

Example 2: The "gorilla" fix by deletion (Google Photos, 2015). Rather than solve the classifier's failure on Black faces at the data and weight layers, Google removed the offending labels ("gorilla," "chimp," "monkey") from the model's vocabulary, and reporting years later found they were still absent (The Verge, 2015; Wired, 2018). This is a real, shipped trade-off: the harm stopped, and the model permanently lost the ability to recognize those animals at all. It is the honest shape of many fixes, a capability traded for a harm avoided. This case is Topic 1.2's deep anchor for bias tracing; here it is only the fix-cost lesson. (see Topic 1.2)

Example 3: The sepsis model and the recall dial (Epic Sepsis Model, external validation, 2021). A widely deployed sepsis-prediction model was found in external validation to miss a majority of sepsis cases while also generating a heavy load of false alerts (Wong et al., JAMA Internal Medicine, 2021). Any fix here is a precision-recall choice with lives and alert-fatigue on the two sides: turn it up and nurses drown in false alarms and start ignoring it, turn it down and more real sepsis slips through. There is no setting that is simply "accurate." The evaluation of this model is Topic 4.6's anchor; here it is the precision-recall trade-off made concrete. Note the second-order cost that a naive average would miss: too many false alarms cause alert fatigue, so nurses begin to ignore the alerts, which silently converts a high-recall setting back into low effective recall. The trade-off is not even stable; pushing one dial too hard can undo itself through human behavior, which is exactly why the cost must be read by real-world consequence and not by the metric alone. (see Topic 4.6)

Example 4: RLHF and the alignment tax (industry practice, established). When model builders fine-tune large language models to be safer and more helpful, they routinely report that the tuning costs some raw capability, a phenomenon widely discussed in the field as the alignment tax, and they publish behavior-change measurements between versions for this reason (see, for example, the documented behavior shifts between GPT model versions studied in Chen, Zaharia, and Zou, "How Is ChatGPT's Behavior Changing over Time?", 2023). The lesson: even the most resourced teams cannot make a model safer for free; they measure the tax and decide it is worth paying. The version-to-version behavior-change record is Topic 10.3's anchor for living model cards. (see Topic 10.3)

Example 5: The fairness dial in credit and hiring (structural). A lender or employer that adjusts a scoring model to equalize approval rates across groups will, by the impossibility results in Section 3D, unequalize some other fairness measure such as the false-negative rate. This is not hypothetical strain: it is the mathematics of any group-scoring system with different base rates. The governance point is that "we made it fair" is never a complete sentence; the complete sentence names which fairness was chosen and which was traded. (The hiring-AI legal exposure this creates is Topic 5.4's anchor. (see Topic 5.4))

Example 6: The security-friendliness trade-off (dealership chatbots, 2023). When businesses relaxed their customer-service bots to be more helpful and open, users manipulated them into absurd commitments, including a Chevrolet dealership bot talked into "agreeing" to sell a car for one dollar (widely reported, December 2023). Every step toward friendlier and less restricted is a step toward easier to manipulate. This adversarial angle is Topic 4.3's anchor; here it is one more face of the same law, that loosening one behavior tightens the risk on another. (see Topic 4.3)

Example 7: The over-correction that made a model unusable (Google Gemini image generation, 2024). A model tuned to increase diversity in generated images over-applied the correction and produced historically inconsistent images, prompting the company to pause the image feature while it re-tuned (widely reported, February 2024). This is the data-and-weight-layer over-correction from Section 3C at full scale: a fix aimed at one real problem (under-representation) pushed the balance so far the other way that the output failed a different test (basic factual plausibility). The lesson is that the failure of a fix is often not "it did nothing" but "it did too much," and only a battery that tests the opposite direction would have caught it before launch. This case is a sibling program's anchor; here it is referenced only for the over-correction lesson. (see Topic 1.2)

Example 8: The version upgrade that quietly changed behavior (measured model drift between versions, 2023). Researchers who tested the same commercial language model across two dated versions found sharp behavior shifts, including a large drop on a task the earlier version handled well (Chen, Zaharia, and Zou, 2023). For a deployer, a vendor's silent "improvement" is a fix someone else made to a model you depend on, and it can regress the exact behavior your product relies on without any change on your side. The defensive move is to run your own regression battery against every new version before adopting it, treating a vendor upgrade with the same suspicion as your own fix. A pre-adoption battery only catches what you tested before the switch, and vendors increasingly roll updates out quietly, without a version number you can pin against; the deeper habit, once you are live, is to keep watching your own product's real outputs after adoption for the same kind of shift, not just once before you flip the switch. This behavior-change record is Topic 10.3's anchor for living model cards. (see Topic 10.3)

Example 9: The filter that was not a fix (content moderation, general pattern). A recurring industry pattern, seen across many chatbots, is to respond to one embarrassing output by adding a keyword filter that blocks that exact phrase, then to be surprised when a reworded prompt sails straight past it. This is the "delete the bad output" non-fix from Section 3B at organizational scale: the filter changes what one string does, not what the model tends to do, so the underlying tendency keeps generating fresh variants the filter never anticipated. Filters are a legitimate outer safety layer, but a team that logs "fixed" after adding one has recorded a bandage as a cure, and its fix-and-tradeoff log will mislead the next person who reads it. The honest entry says "added a filter for symptom X; root cause in the model remains; re-test on paraphrases."

Where people go wrong

  • "We just need to fix that one behavior." There is no "that one behavior" to fix in isolation. Because the model is one coupled system (Section 3B), you change data, weights, or instructions and the effects spread. The honest framing is never "fix behavior X" but "change the model to improve X and find out what else moved."
  • "A system-prompt tweak is safe because it is small." Small change and small risk are different things. The system-prompt layer is the least tested because it ships fastest, and it sits in front of every single interaction, so its blast radius is the widest. The Grok incident was one instruction (xAI, July 2025). Treat prompt changes as the highest-risk fixes to test, not the lowest.
  • "The fix worked, we tested it." Testing that the fix improved its target tells you almost nothing. The fix nearly always improves its target. The question is what happened to everything you were not looking at. A regression battery that only contains the problem you fixed is a battery designed to reassure you, not inform you.
  • "There must be a fix with no downside if we are clever enough." For a large class of problems there is not, and this is proven, not merely difficult. The fairness impossibility results (Kleinberg et al., 2016; Chouldechova, 2017) and the no-free-lunch theorem (Wolpert and Macready, 1997) mean some trade-offs can only be chosen, never removed. Chasing the free fix wastes time you should spend choosing and documenting the trade-off.
  • "Adding more training data can only help." Adding data to strengthen a weak case can dilute cases that already worked, or overcorrect so the model now over-predicts the class you reinforced. Data-layer fixes are the most durable and also carry their own regressions; they are not a safe default that avoids trade-offs.
  • "If we make it fairer on one measure, it is fairer." Fairness is not one measure. Improving equal approval rates can worsen equal error rates. "We made it fair" is an incomplete sentence; the complete one names which definition you honored and which you traded, and why.
  • "Regression is the same as drift." No. A regression is a failure you caused by changing the model, visible immediately. Drift is a failure the changing world caused while the model sat still, visible over time (Topic 4.5). Confusing them sends you looking for a code change when the world moved, or blaming the world when you shipped a bad fix. (see Topic 4.5)
  • "Documenting the change is bureaucracy." The fix-and-tradeoff log is the difference between a fix you can defend and a thing that happened to your model. When harm surfaces, the log is the only record of who saw the cost, measured it, and accepted it. Without it there is nothing to inspect and no one to hold, which is the opposite of governance.
  • "We can always roll it back if it breaks." Rollback recovers the model, not the world. If a bad fix reached users, produced harmful output, or made real decisions before you noticed, reverting the model does not un-send the outputs. Irreversibility of the consequences, not of the code, is why you test before shipping, not after.
  • "A vendor's upgrade is the vendor's problem, not mine." When you deploy a model you did not build, the vendor's fixes are fixes to your system that you did not make and cannot see. A silent version change can regress the exact behavior your product depends on. Treating a vendor upgrade as someone else's concern means you find out about its cost from your own users. The defensive habit is to run your own regression battery against every new version before you adopt it. (see Topic 10.3)
  • "If the model is better on average, the fix was good." An average can rise while the cases that matter most fall. A fix that lifts overall accuracy by helping many easy cases, while quietly regressing a small number of high-stakes cases, is often a bad fix, not a good one. Governance cares about where the cost landed and on whom, not only about the mean. Read the trade-off by group and by stakes, never only by the headline number. A public benchmark score or a vendor's own eval dashboard is a headline number of exactly this kind: it tells you the model improved on whatever that benchmark measures, and nothing about the case near you that the benchmark never sampled. Treat any benchmark or leaderboard claim as a reason to build your own battery, never as a substitute for one.

Questions people ask

What is model (machine-learning model)?
A system that has learned patterns from data by adjusting internal numbers (weights) so its outputs fit that data. For a large language model, the deployed system is the trained weights plus a layer of instructions wrapped around them.
What is coupled system?
A system whose parts are interdependent so that changing one part affects others. A machine-learning model is coupled because its behaviors are spread across shared weights, which is why there is no local edit.
What is distributed representation?
The property that a model's knowledge and behaviors are spread across many weights rather than stored in one place. It is what lets a model generalize, and also why you cannot turn one behavior up without moving others.
What is fix (in this topic)?
Any deliberate change to a model intended to improve a behavior, made at one of three layers: the training data, the model weights (retraining or fine-tuning), or the system prompt (instructions).
What is layer of a fix?
Where a change is made. Data layer (the examples the model learns from), weight layer (the tuning itself), or instruction layer (the system prompt on top). Each layer ships at a different speed and tends to break something different.

Keep going

Watch the lesson and prove it

Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.

Watch the lesson and prove it →