Skip to main content

Train a model with your own hands and watch what it actually learns

The short answer

A model learns exactly what its data and objective make it learn

There is no comprehension in training, only pattern-fitting toward the number you set as the goal. This single sentence explains Tay, the snow detector, and every model you will ever govern. Master it and the rest follows.

What you will be able to do

  • Train a simple classification model yourself, from data you choose, using a free tool at your own desk, and observe its predictions change as the training data changes.
  • Explain in plain language what "training a model" actually does: it fits a function to patterns in data, optimizing an objective you set, with no understanding of your intent.
  • Distinguish what you intended to teach a model from what it actually learned, and name at least three ways those two things come apart (shortcut learning, skewed data, a mismatched objective).
  • Demonstrate the core lesson of the Microsoft Tay case by deliberately feeding your own model skewed data and watching it inherit the skew.
  • Produce the first entry in a "how my model fails" record: a written, honest account of one thing your model learned that you did not intend.
  • Apply the phrase "a model learns exactly what its data and objective make it learn" to a real system you are responsible for, and locate where its data and objective actually come from.
  • Justify, to a skeptical colleague, why building and breaking a model yourself is a prerequisite for governing one, not an optional detour.
  • Convert an impressive accuracy claim into a specific list of evidence you would demand before trusting it, drawn from the four ways intent and reality split.
  • Reframe the phrase "the AI decided" into an accountable chain of human choices about data, objective, and deployment.

The ideas, one by one

The data is the program

In machine learning, choosing the training data is the closest thing to writing the source code. Whoever chooses the data, or fails to examine it, determines what the model becomes and owns the result.

The objective is a wish taken literally

A model satisfies the exact metric you gave it, in whatever way is cheapest, including ways you would be ashamed of. Always ask how a machine could make your objective look good while betraying its intent.

A model has no values; the values are in the choices

Tay did not choose hatred; it reflected the data it was fed. Accountability belongs to the human choices about data, objective, and deployment, not to "the AI." Precise language is the first governance control.

Good scores hide bad models

Shortcut learning, skewed data, and mismatched metrics all produce excellent numbers on the maker's own test. Accuracy is a question, not an answer; the real work is designing tests that expose the four intent-versus-reality splits.

Intent and reality split in four recurring shapes

Shortcut learning, skewed data, mismatched objective, and drift. Name them, and you know what to test for on any system, whether you built it or not.

Building first is what lets you see

Having trained a model yourself converts read facts into known facts: the data is the program, the objective is literal, a decision is not being made. That knowledge is exactly the gap that vendors and internal teams rely on you not having.

The lesson scales without changing

A desk classifier and a frontier model fit patterns in data to satisfy an objective in the same way. The small model is a microscope on a mechanism that is identical all the way up.

Governing a model is a standing commitment, not a one-time review

The world drifts and a model keeps answering as if the past were still true. The trained model you start here is watched, tested, and re-examined through the whole program for exactly this reason.

This build is the foundation of your dossier

The trained model plus its "how it fails" record is the artifact that grounds your evaluation report, your conformity file, and your defense against a hostile board. Every credible answer you give later traces back to having done this.

Not choosing is a choice

Pointing a model at convenient historical data and picking an easy-to-measure objective are decisions, even when nobody makes them deliberately. "We did not choose anything problematic" is not a defense; examining the defaulted data and objective, and forcing a real decision where a default was made silently, is the work.

Automation is consistency, not objectivity

A model applies a frozen set of past human judgments uniformly and fast. If those judgments were skewed, automation scales the skew evenly to everyone. Ask who the errors land on, not only how many there are.

The two questions are fixed for life

For any model, claim, or proposal, ask first what the data really is and what the objective exactly is. Everything else in this program sharpens what you do with the answers; these two questions never change.

Two models agreeing is not proof

Models trained on the same data inherit the same flaws and will be confidently wrong together. Confirmation has to come from data or tests they did not share, not from a second model nodding along.

Passing a test proves only what the test checked

The dangerous failures hide from tests written by the model's own builders. Trust comes from adversarial tests aimed at the failure modes you can name, which is why this program tests for the four splits by design.

Listen to it

The same lesson, talked through by two hosts as episode 5 of the podcast.

Real cases

These examples show "a model learns exactly what its data and objective make it learn" across different domains, regions, and years. Each is attributed. Where a case is owned in depth by a later topic, it is pointer-referenced, not retold here.

Example 1: Microsoft Tay, United States, 2016 (the anchor). Microsoft released Tay as a Twitter chatbot designed to learn conversational style from the public. A coordinated group exploited its imitation behavior, including a "repeat after me" function, and fed it hateful content.

Within about sixteen hours Tay was posting racist and antisemitic messages and Microsoft took it down (The Verge, 24 March 2016). The teachable core: the model was not misprogrammed. It learned its inputs faithfully, at scale and speed, with no internal check on the content of what it learned.

Tay is the clearest possible demonstration that a learning system's outputs are its data's values, not its makers' intentions.

Example 2: The husky-versus-wolf snow detector, research setting, 2016. In a study of model explanations, researchers built an image classifier that appeared to distinguish wolves from huskies and then showed, using an explanation tool, that it was largely relying on snowy backgrounds rather than the animals (Ribeiro, Singh, and Guestrin, KDD 2016).

No malice, no bad data collection scandal, just a model that found the cheapest pattern that separated the piles it was given. This is shortcut learning in its purest laboratory form and it is the case to keep in mind whenever a model's accuracy looks too good.

Example 3: Chest x-ray models that learned the hospital, not the disease, 2018. A research team evaluated pneumonia-detection models across multiple hospital systems and found the models partly keyed on site-specific image features (effectively, which scanner and which hospital produced the image) and generalized worse across sites than their internal scores suggested (Zech et al., PLOS Medicine, 2018).

A model can post strong numbers in the building where it was trained and quietly fail in the building next door, because it learned a shortcut correlated with the outcome rather than the outcome's real cause. For a governance leader, this is the case that kills "but the accuracy was ninety-plus percent" as a defense.

Example 4: Google Photos label failure, United States, 2015 (pointer). An image classifier attached a grossly offensive animal label to photos of Black users, a harm traced directly to what the training data did and did not contain.

This case is owned in depth by Topic 1.2, which traces a bad prediction back to its data. (see Topic 1.2) It is named here only to show that "the model learned what its data encoded" and "the data did not represent everyone" are the same failure viewed from two angles.

Example 5: Speech recognition that heard some voices worse than others, United States, 2020. A peer-reviewed study tested five major commercial speech-recognition systems (from Amazon, Apple, Google, IBM, and Microsoft) on recorded interviews and found the systems misrecognized the speech of Black speakers at nearly twice the word-error rate of white speakers, a gap the authors traced substantially to acoustic training data that underrepresented Black speakers' speech (Koenecke et al., "Racial disparities in automated speech recognition," Proceedings of the National Academy of Sciences, 2020).

No engineer set out to build a system that worked worse for one group. The training data was skewed, so the models learned the voices they heard most and stumbled on the ones they heard least. This is Split 2, skewed data, at industrial scale, and it is a case to remember whenever someone claims a model "works well" without asking "works well for whom."

Example 6: The everyday version, any organization, now. The examples above are famous because they were caught. The common case is the one nobody writes about: a support-ticket classifier that routes complaints, trained on a year of tickets a company happened to have, quietly learning that tickets mentioning a certain product line are "low priority" because that is how they were historically handled, and continuing to bury them long after the company decided that product mattered.

No headline, no scandal, just a model faithfully carrying forward a pattern from data that nobody re-examined. Most of the models you will govern are this kind. The lesson from Tay applies to all of them: the model learned exactly what it was fed, and someone chose what to feed it.

Example 7: The objective that quietly optimized the wrong thing, Meta (Facebook), United States, 2018 to 2021. Internal Facebook research, made public through disclosures from whistleblower Frances Haugen and reported by The Washington Post, showed that starting in 2017 the News Feed ranking model, alongside a 2018 change meant to boost "meaningful social interactions," weighted a reaction emoji (including the angry reaction) roughly five times as heavily as a plain "like."

The model, chasing that objective, learned to rank content that provoked anger more highly, and internal researchers linked the change to more divisive and lower-quality content in people's feeds; the company later reduced the weight (Merrill and Oremus, "Five points for anger, one for a 'like': How Facebook's formula fostered rage and misinformation," The Washington Post, 26 October 2021, reporting on documents from the Haugen disclosures).

No team wrote "promote outrage" into the objective; they wrote a ranking objective meant to reward engagement, and the model found that anger maximized it. This is Split 3, a mismatched objective, and reward hacking in production: the measure (interaction-weighted engagement) became the target and stopped being a good stand-in for the goal (value to the user).

For a governance leader it is the reminder that the objective, not just the data, is a lever that can produce harm no one intended.

What the seven examples share. Across a chatbot, a research classifier, a medical model, a photo tagger, five speech systems, a mundane ticket router, and a recommender, the mechanism is identical: the model learned exactly what its data and objective made it learn, and the harm lived in the gap between what its makers intended and what the data and objective actually encoded.

None of these failures required a bad actor or a coding error. Each required only that someone trust the model without having examined its data and its objective, and without having built the kind of test that would expose the gap.

That is the failure this topic exists to make you incapable of committing.

Where people go wrong

  • "The model became biased / racist / cruel." A model does not become anything, because it has no values to change. It reflects its data and satisfies its objective. Saying Tay "became racist" hides the real cause: it was fed hateful data and told to imitate. Precise language matters, because "the model became biased" points at the model, while "the data was skewed and the objective rewarded imitation" points at the two things you can actually govern.
  • "If we didn't program the bad behavior, we aren't responsible for it." You chose the data and you chose the objective, and those choices produced the behavior as surely as a written rule would have. Microsoft did not write racist code, and Microsoft was still responsible for Tay. In machine learning, the data is the program. Choosing it, or failing to examine it, is an act you own.
  • "High accuracy means the model is good." Accuracy on the test its makers ran can hide shortcut learning (the snow detector), skewed data, and a badly chosen metric. A model that always predicts the majority class can score ninety-five percent on skewed data and be useless. Accuracy is a starting question, never a conclusion. The real question is what the number would do under a test designed to expose the four splits.
  • "We can just fix the bad outputs after we see them." Sometimes, but every fix touches the same shared machinery and can break something you were not watching. Filtering Tay's worst words would not have changed what it learned, only what it said out loud. Fixes are never free, and the reasons why are the whole of Topic 1.3. (see Topic 1.3)
  • "You need to be a data scientist to understand this." You need to train one small model once. The mechanism (data in, objective set, adjust to shrink loss) is understandable by anyone, and you will do it yourself in Section 7 with no code required. What you do not need is to be able to build a production system. What you do need is to have watched a model learn, so that vendor claims and internal proposals meet a mind that knows what learning actually is.
  • "The vendor trained it, so the vendor understands what it learned." Often the vendor knows their accuracy number and little else about the specific patterns their model relies on. Shortcut learning is usually discovered later, by someone who went looking. Assuming the builder understands the failure modes is exactly the assumption that lets failure modes survive into deployment. Your job is to make someone go looking, and to know what looking means.
  • "Watching one small model teaches me nothing about a huge one." The scale differs; the mechanism does not. A frontier model and your desk classifier both fit patterns in data to satisfy an objective, and both will faithfully carry forward whatever the data and objective encode. The small model is a microscope, not a miniature. It shows you the mechanism at a size you can hold, and the mechanism is the same all the way up.
  • "Once trained, the model is fixed and I can reason about it once." The world drifts, and a model trained on the past keeps answering as if the past were still true. A model that was fair and accurate at launch can become neither, without one line of code changing, because the data around it moved. Governing a model is not a one-time act; it is a standing commitment, which is why later modules build logging, evaluation, and drift monitoring around the model you are starting to build now.
  • "Automating the decision makes it more objective." Automation makes a decision faster and more consistent, not more objective. A model freezes a set of past human judgments (in the data and the objective) into a form that is applied uniformly and is hard to question. If those frozen judgments were skewed, automation applies the skew consistently to everyone, which can be worse than inconsistent human bias, not better, because there is no off day and no second thought. Consistency is not the same as fairness.
  • "If the model is right most of the time, the rare errors do not matter much." It depends entirely on which cases are the errors and what they cost. A model that is right ninety-five percent of the time but wrong on exactly the vulnerable minority, or on exactly the high-stakes cases, can be far more harmful than its overall score suggests. Aggregate accuracy averages away the pattern of who gets hurt. Always ask where the errors land, not just how many there are.
  • "We can trust the model because we trust the people who built it." Trust in people is not evidence about a model. Well-intentioned, competent teams ship models with shortcuts and skews all the time, precisely because those failures are invisible from the inside and only appear under tests designed to find them. Trust has to be earned by the model, through evidence, not inherited from the reputation of its builders. This is the adversarial-by-default posture the whole program runs on.
  • "A model that improves with use is self-correcting." Improving with use means becoming more like recent inputs, which is only good if those inputs are good. Tay improved with use, straight into hate. A system that learns continuously needs its inputs governed continuously; "self-correcting" is a marketing frame for "absorbs whatever it is fed," and absorbing is not the same as correcting.
  • "If two models give the same answer, the answer must be right." Two models trained on the same skewed data will agree confidently and be wrong together, because they inherited the same flaw. Agreement between models is not independent confirmation when they share training data or assumptions; it can simply be the same error, twice. Independent evidence has to come from data or tests the models did not share.
  • "The training data is the vendor's problem, not ours." Once you deploy a model, its learned behavior becomes your organization's behavior in front of your users and regulators. "The vendor trained it" does not transfer the consequences of what it learned; it only means you must demand visibility into the data and objective you are now accountable for. Outsourcing the build does not outsource the responsibility.
  • "We tested it and it passed, so it is safe to deploy." It passed the test you thought to run. Shortcut learning, skew, and drift hide from tests that were not designed to expose them, which is most tests written by the same people who built the model. Passing is only as strong as the test, and the test is only as strong as the failure modes you imagined; adversarial testing exists precisely because the dangerous failures are the ones you did not think to check.

Questions people ask

What is model (machine learning model)?
A program whose behavior is determined not by hand-written rules but by patterns fit to data during training. A trained model takes an input and produces an output (a prediction, score, or label) based on the patterns it absorbed. It has no understanding, intent, or values; it reflects its training data and the objective it was optimized for.
What is training?
The process of showing a model many examples paired with desired answers and repeatedly adjusting the model's internal settings so its answers on those examples grow closer to the desired ones. Training does not teach a concept; it fits the cheapest available pattern in the supplied data that satisfies the objective. More on Training
What is training data?
The set of labeled examples a model learns from. In machine learning the training data functions as the program: it determines what the model becomes, so choosing it and examining it are core governance acts. Skewed or unrepresentative training data produces a model that faithfully carries the skew forward. More on Training data
What is label?
The desired answer attached to a training example (for instance "cat" on a cat photo, or "warm" on a customer message). The pattern of labels across the data defines what the model is being asked to reproduce.
What is objective (loss function)?
The single quantity that training tries to make small, encoding "what counts as good." A model satisfies its objective literally and in whatever way is cheapest, including ways its makers would reject, so a poorly chosen objective is a common source of unintended behavior.

Keep going

Watch the lesson and prove it

Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.

Watch the lesson and prove it →