Skip to main content

Model cards and system cards for your own systems, written so an outsider could act on them

The short answer

Two cards, two jobs

A model card documents the trained model, its capabilities, training, and measured performance. A system card documents the deployed system, the model plus the application layer of prompts, retrieval, filters, and human review. If you deployed anything, you owe both, because the gap between what a model can do and what a system actually does is where the real risk lives.

What you will be able to do

  • Write a model card for a model you built, using the structure introduced by Mitchell and colleagues (2019), covering model details, intended use, out-of-scope use, factors, metrics, evaluation and training data, disaggregated results, ethical considerations, and caveats.
  • Write a system card for a deployed system, covering the components, data flows, human oversight points, system-level evaluation, red-team findings, deployment boundaries, monitoring, and the update process, so the application layer is documented, not just the model.
  • Distinguish a model card from a system card and explain why the gap between what a model can do and what a deployed system actually does is where most real-world failures live.
  • Apply the outsider test to any card: name the specific outside reader, name the decision they must make, and check that the card lets them make it without you present.
  • Connect your cards to the obligations that now demand them, including EU AI Act Article 53 documentation for general-purpose AI model providers (Annex XI and Annex XII), the NIST AI Risk Management Framework Map and Measure functions, and ISO/IEC 42001, without overclaiming what any one document satisfies.
  • Explain why a card is a living document that must be versioned and dated, using the behavior-drift evidence, and design a card so that staleness is visible rather than hidden.
  • Assemble a card from evidence you already produced (your "how my model fails" explanation, your evaluation report, your data provenance file, your incident log) rather than writing fresh prose, and hand the finished cards to your evidence annex.
  • Distinguish real oversight from nominal oversight in a system card, and document a human review step only after testing whether the human can actually see and act before harm occurs.
  • Split ownership correctly when you did not train the model: write a system card for your deployment, treat a vendor's model card as a cited dependency, write a fresh model card when you fine-tune, and record any documentation gap as a named supplier risk.

The ideas, one by one

The outsider test is the only standard

A card earns its existence only if a named outside reader can make a real decision from it without you in the room. Name the reader, name the decision, and strike every sentence that does not serve it. This one test kills marketing language, vagueness, and hidden failure modes at once.

Honesty is load-bearing

The most valuable content in any card is the honest bad news: out-of-scope uses, failure modes, and the groups or conditions where performance drops. A caveat you wrote is evidence of diligence; the identical fact discovered by an auditor is evidence of concealment. Authorship is the whole difference.

Disaggregate or mislead

A single overall accuracy number is exactly the failure model cards were invented to fix. Report performance broken down across the factors you named, because a model strong on average can be dangerously weak for one group or one condition.

A card must be living

Dated, versioned, and carrying a changelog and review triggers. The behavior-drift study shows what an un-versioned system costs: a product changed under its users and no document recorded it. An undated card is worse than no card, because it carries your authority while telling a stale lie.

Assemble, do not invent

A good card is built from evidence you already produced: your failure explanation, your provenance file, your evaluation report, your incident log. If you are writing a claim with no artifact behind it, you have found either an unsupported claim or a hole in your evidence work.

One artifact, several regimes

Written well, your cards do real work against the EU AI Act's general-purpose AI model documentation duties (Article 53, Annex XI and XII), high-risk documentation (Articles 11 and 13), the NIST AI RMF Map and Measure functions, and ISO/IEC 42001. Do not overclaim: no single card makes you compliant, but one maintained card set beats a drawer of contradictory compliance PDFs.

A missing vendor card is a finding

If you build around someone else's model and cannot get honest documentation of its capabilities and limitations, that gap belongs in your system card as a named supplier risk, not hidden. The EU AI Act now gives you standing to demand it.

A card is a summary, not a report

The evaluation report substantiates; the card communicates. A good card fits on a page or two and points to the underlying artifacts for the detail. Its value is not length but the speed with which an outsider can reach the right decision. If your card is a thirty-page document, you have written a report and mislabeled it.

Document oversight honestly, not aspirationally

A human review step listed as a control is worthless if the human cannot see or act before harm occurs. Distinguish real oversight from nominal oversight in the system card, and write the nominal ones down as gaps. An oversight control that cannot fire in time is not protection, it is the appearance of protection.

The card is a rehearsal for the audit

The self-attack you run on your own cards is the same read a hostile board or examiner will run in Module 13. The better your cards survive your own worst reading now, the fewer surprises the real adversary finds later, and the more the auditor trusts everything behind them.

A narrow claim you can prove beats a wide one you cannot

Broad intended-use statements sound capable and collapse under audit. Narrow the intended use until it is testable, then defend it. The reader does not need your system to sound impressive; they need to know exactly where your evidence holds and where it stops.

Ownership splits, responsibility does not

When you integrate a vendor model you still owe a system card, when you fine-tune you owe a fresh model card, and a missing vendor document is your finding to record, not your excuse to skip. You can cite a dependency, but you cannot outsource the description of the system you built and serve.

Listen to it

The same lesson, talked through by two hosts as episode 78 of the podcast.

Real cases

These examples show model and system documentation working, and failing, in real settings. Details are attributed; where a specific figure would be invented, it is left out.

Example 1: The behavior-drift study (the anchor). Chen, Zaharia, and Zou compared the March 2023 and June 2023 releases of GPT-4 and GPT-3.5 on tasks including math, sensitive questions, code generation, and visual reasoning ("How Is ChatGPT's Behavior Changing over Time?", arXiv:2307.09009, 2023). They reported large swings, including GPT-4's accuracy on identifying prime numbers falling from about 84 percent to about 51 percent between the two versions. Other researchers, including Arvind Narayanan and Sayash Kapoor, noted that part of the effect reflected a change in the model's answer formatting rather than pure capability loss, and that the test set structure amplified the appearance of collapse. Both things are true, and the governance lesson holds regardless of which interpretation you favor: the deployed behavior of a named product changed under its users, no living document recorded the change, and the users had to reverse-engineer it after the fact. A versioned system card with a changelog is the artifact whose absence made the whole episode possible.

Example 2: Google's model card practice. Following the 2019 proposal, Google published model cards for some released models, including face detection and object detection models, presenting intended use, limitations, and performance disaggregated across groups such as skin tone and age. The value was not the numbers looking good. It was that an outsider could see, before integrating, that performance varied across groups and could decide accordingly. That is the outsider test passing: the card changed a downstream decision.

Example 3: OpenAI and Anthropic system cards. OpenAI's GPT-4 System Card (March 2023) and subsequent system cards for later models documented red-team findings, categories of risk, mitigations applied, and residual risk after mitigation. Anthropic has published model cards for its Claude models covering intended uses, evaluations, and safety testing. The pattern that matters for you: these documents state the bad news, the things the system can still be made to do wrong after safety work, because a system card that only listed successes would tell an integrator nothing they could act on.

Example 4: Hugging Face and the documentation-by-default move. The Hugging Face model hub attaches a model card (a structured README with a standardized metadata block) to hosted models, and prompts uploaders to fill in intended use, limitations, bias, and training data. Millions of models carry at least a skeleton card as a result. The lesson is about defaults: when the card is part of the publishing workflow rather than an afterthought, documentation coverage rises sharply. When you build your own internal model registry, make the card a required field, not an optional one.

Example 5: The vendor who would not give you a card. A common real situation, drawn from procurement practice rather than a single named incident: an organization wants to deploy a general-purpose AI model from a vendor into a customer-facing system, and asks for the model's documentation of capabilities, limitations, and training-data summary. The vendor declines, citing trade secrets. Under the EU AI Act, a general-purpose AI model provider owes downstream integrators the Annex XII information (while protecting genuine trade secrets), and a provider of a systemic-risk model owes more. The absence of a usable model card is itself a governance finding: it means you cannot write an honest system card, because a component of your system is undocumented. The correct move is to record that gap in your own system card as a known limitation and a supplier risk, not to paper over it.

Example 6: The medical AI with a narrow validation and a broad deployment. Across health-AI literature, a recurring failure pattern is a model validated on data from a small number of hospitals or one imaging device, then deployed across many sites with different populations and different equipment. The model card's factors and evaluation-data sections exist precisely to catch this: if the card honestly states "validated on data from three hospitals using one manufacturer's scanner," a downstream site with a different scanner and population knows immediately that its use is outside the validated conditions. When that line is missing from the card, the mismatch is discovered only after the errors accumulate on real patients. The card is the difference between a boundary that is stated and a boundary that is crossed blindly.

Example 7: The EU AI Act turning cards into law. The move from voluntary practice to legal obligation is not hypothetical. The EU AI Act's general-purpose AI model rules require providers to draw up and keep up to date technical documentation (Annex XI) and to give downstream integrators documentation of the model's capabilities and limitations (Annex XII), with these obligations applying from 2 August 2025 (Regulation (EU) 2024/1689). In substance, the law now mandates a model card for general-purpose models. The lesson for you is that building the card habit before you are legally required to is far cheaper than reconstructing documentation under enforcement pressure, and organizations that already document by default absorbed this obligation as routine rather than as a scramble.

Example 8: The disaggregated audit that a card would have pre-empted. New York City's Local Law 144 requires an annual independent bias audit of automated employment decision tools, with the results posted publicly (a 2021 law, in effect since January 2023 and enforced from July 2023). The published audits are, in effect, forced disaggregated results: selection rates broken down by sex and by race-ethnicity groups. Where employers had already maintained honest model cards with disaggregated performance, the audit confirmed what they already knew and documented. Where they had not, the audit was the first time anyone saw the group-level numbers, sometimes uncomfortably in public. The card is the difference between finding your own disaggregated results in private, on your schedule, and having a mandatory audit surface them for you.

Example 9: The changelog that turned a scramble into a routine. Consider the contrast the behavior-drift study sets up, applied to a well-run team. When a provider ships a new model version accompanied by a system card update and a changelog that says, in plain words, "arithmetic and formatting behavior changed; see revised metrics," every downstream integrator receives a signal to re-test the specific behaviors that moved. The change is still disruptive, but it is a managed disruption: integrators know what to check and when. The 2023 episode was the opposite, a silent change discovered by outside researchers months later. The lesson is not that models should never change, they must, but that a change without a changelog converts a routine version bump into a forensic investigation for everyone downstream. The changelog is cheap; its absence is what makes the drift expensive.

Where people go wrong

  • "A model card and a system card are the same thing." They are not. The model card documents the trained model in isolation, its capabilities, training, and measured performance. The system card documents the deployed system, the model plus the prompt, retrieval, filters, human review, and deployment context. Most real harm comes from the application layer, which the model card cannot describe. If you deployed anything, you owe both.
  • "The card is a marketing document." The fastest way to make a card worthless is to write it to impress. A card exists to help an outsider make a decision, and marketing language helps no one decide anything. If a sentence would fit in a sales deck, it probably fails the outsider test. Comms can write the brochure. Engineering and governance write the card.
  • "We can skip the out-of-scope section." The out-of-scope section is the part with the most governance value, because a model safe for one use is often dangerous for another. Omitting it does not make the dangerous uses safe, it just means an integrator finds the boundary by crossing it. Naming out-of-scope uses is how you keep your model from being deployed into a context you never validated.
  • "Overall accuracy is enough." A single aggregate number is exactly what model cards were invented to fix. A model that is 90 percent accurate overall can be 60 percent accurate for one group or one condition. Report performance disaggregated across the factors you named. The number that embarrasses you is the one the card exists to carry.
  • "We wrote it once, so we are done." A card that stops matching the system is worse than no card, because it carries your authority while telling a lie. Systems change: models get retrained, vendors push updates, retrieval indexes shift, prompts get edited. Every card needs a date, a version, a changelog, and a review-by trigger list. The behavior-drift study is the standing proof that "once" is not enough.
  • "Trade secrets mean we cannot document anything." Documentation and confidentiality coexist. You can state capabilities, limitations, intended use, evaluation conditions, and known failure modes without exposing proprietary weights or training internals. The EU AI Act's downstream-documentation duty is explicitly framed to protect trade secrets while still requiring enough for integrators to understand capabilities and limitations. "Trade secret" is not a valid reason to ship a card with no out-of-scope section.
  • "If we do not write the bad news, it is not there." The failure mode exists whether or not you document it. Not writing it down does not remove it; it just guarantees that whoever finds it, an auditor, an incident responder, a harmed user, finds it without your framing and without evidence that you looked. A caveat you wrote is diligence. The same fact discovered by someone else is concealment.
  • "Our vendor's model card is our problem solved." If you deploy a system around someone else's model, their model card is a dependency of your system card, not a substitute for it. You still owe a system card for what you built around their model, and if their model card is missing or thin, that gap belongs in your system card as a named supplier risk.
  • "Benchmark numbers speak for themselves." A metric with no evaluation conditions is close to meaningless. A number needs the test set, the date, the version, and the definition of the metric attached, or a reader cannot tell whether it applies to their situation. An impressive number on an unstated benchmark is a red flag to a good auditor, not a reassurance.
  • "A longer card is a better card." Length is not thoroughness. A thirty-page card is usually a report wearing a card's name, and no outsider will read it in the moment they need to decide. The card is the summary that points to the underlying artifacts; the detail lives in the evaluation report, the provenance file, and the logs. Optimize the card for the two-minute decision, not for the appearance of effort.
  • "We can copy last quarter's card for the new version." Copying an old card onto a new model version and changing the date is the fastest way to publish a confident lie. If the model changed, the metrics, factors, and failure modes may all have changed with it, and the behavior-drift study is the standing proof that they can move sharply. A new version demands re-evaluation and a genuine changelog entry, not a find-and-replace on the version number.
  • "The card is the governance team's job, not engineering's." A card written by people who did not build or evaluate the system will be vague where it matters, because the load-bearing content (real failure modes, actual evaluation conditions, the honest oversight gaps) lives in the heads and artifacts of the people who did the work. Governance can own the structure and the outsider test; engineering must supply the substance. A card written entirely by one side without the other is either accurate but unreadable or readable but wrong.

Questions people ask

What is model card?
A short, structured document that travels with a trained model and states, in plain language, what the model is, what it is for, where it works, where it fails, and how that is known. Introduced formally by Mitchell and colleagues (2019). Covers model details, intended and out-of-scope use, factors, metrics, evaluation and training data, disaggregated results, and caveats. More on Model card
What is system card?
A document describing a deployed AI system rather than a single model: the model or models plus the application layer of prompts, retrieval, filters, human oversight, and deployment context. It documents what the deployed system actually does, which is a narrower and different thing from what the underlying model can do. More on System card
What is application layer?
Everything wrapped around a model in a deployed system: system prompts or instructions, retrieval, input and output filters, human review steps, rate limits, and usage policy. The application layer determines the gap between a model's raw capabilities and the deployed system's actual behavior, and it is documented by the system card, not the model card.
What is outsider test?
The standard by which a card is judged. Name the specific outside reader (integrator, auditor, incident responder, buyer, or future maintainer), name the decision they must make, and check that the card lets them make it correctly without the author present. A sentence that helps no outsider decide anything does not belong in the card. More on Outsider test
What is out-of-scope use?
The uses, contexts, populations, and conditions for which a model or system was not validated and should not be relied on. The out-of-scope statement is the highest-governance-value section of a card, because a model safe for one use is often dangerous for another.

Keep going

Watch the lesson and prove it

Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.

Watch the lesson and prove it →