The Model Inventory and Tiering That Survives an Examiner
The short answer
An examiner asks for the inventory before any single evaluation report
One strong report proves one system was checked. A complete, tiered inventory proves the organization knows what it runs at all, which is the artifact Earnest Operations could not produce after eleven years.
What you will be able to do
- Analyze why a model inventory, not a single system's evaluation report, is the first artifact an examiner or auditor asks for, using the Earnest Operations settlement as the case where its absence became a legal finding.
- Define the fields a defensible inventory row must carry: a named owner, a stated purpose, a materiality score, a complexity score, a resulting tier, a current validation status, and a next review date.
- Apply a two-factor tiering rule, materiality by complexity, to sort a mixed set of real systems into three tiers, and justify each placement in one sentence.
- Distinguish your organization's own materiality-and-complexity tier from a legal risk category such as the EU AI Act's high-risk classification, and explain why the two exist for different reasons and can disagree.
- Diagnose the shadow-model and undocumented-override problem: models and human bypass patterns that exist in production but never made the inventory, and explain why an inventory with silent gaps is worse than no inventory, because it looks complete.
- Argue a tier up or down for a specific model when its stated risk framing (for example, "it only drafts, a human decides") understates the harm a wrong output could still cause.
- Assign validation depth, revalidation frequency, and sign-off authority to each of the three tiers, and explain why the current US banking model risk guidance excludes generative and agentic AI from its own scope, and what that means for who must extend the discipline anyway.
- Critique an inventory handed to you by a colleague or a vendor, identifying missing fields, silently excluded systems, and tier assignments that do not match the rule applied to the rest of the list.
- Defend a completed model inventory against a hostile examiner's core question: how do you know this list is the whole list.
The ideas, one by one
A defensible inventory row has seven fields
Name and version, owner, purpose, materiality score, complexity score, resulting tier, and validation status with a next review date. A row missing any of these cannot be defended under examination.
The tiering rule is materiality by complexity, summed
Materiality (1 to 3) scores the worst realistic outcome if the system is wrong. Complexity (1 to 3) scores how traceable and how variable the system's output is. The sum, 5 to 6, 3 to 4, or 2, produces Tier 1 Critical, Tier 2 Significant, or Tier 3 Limited, mechanically, without room for a preferred outcome to bend the score.
A human in the loop does not lower materiality by itself
Review quality degrades under volume the same way clinician attention degrades under a flood of false alarms. Score the worst realistic outcome regardless of who technically clicks the button, which is exactly why a generative customer-reply assistant can land in Tier 1 despite making no decision on its own.
The inventory is not a legal risk classification
(see Topic 5.3) The EU AI Act's high-risk category and this topic's materiality-and-complexity tier answer different questions and can disagree; a system can be legally low-risk and organizationally Tier 1, or the reverse.
A regulatory carve-out is not evidence of low risk
SR 26-2 expressly excludes generative and agentic AI from its scope. The discipline this topic teaches, inventory, tiering, validation, review dates, transfers to those systems as a matter of practice, whether or not a specific rule currently requires it.
Shadow models and undocumented overrides are the two ways an inventory quietly fails
A shadow model never entered the list at all. An undocumented override means the model's stated decision and its actual real-world outcome are two different, unreconciled things, exactly what the Earnest investigation surfaced.
Each tier's requirements are non-negotiable and deliberately unequal
Tier 1 gets a full evaluation report, independent challenge, and annual revalidation. Tier 3 gets basic documentation and a review every few years. Treating every model the same, whether by over-scrutinizing or under-scrutinizing, defeats the purpose of tiering at all.
The inventory decays without active maintenance
Overdue review dates, deleted rather than retired rows, and version changes that never reopen a row all quietly turn a once-accurate inventory into a document the organization treats as current long after it stops being true.
The inventory travels forward
This artifact feeds the conformity file, the evidence annex, and every model and system card the organization produces. Build it complete now and every downstream artifact that depends on it inherits that completeness. (see Topic 5.6) (see Topic 10.2) (see Topic 10.3)
Tiering errors run in two directions, and both are costly
Under-tiering hides real risk behind a lighter review than a system's true consequence warrants. Over-tiering makes the inventory unaffordable to maintain and dilutes the meaning of the highest tier. Both come from letting a preferred outcome, rather than the rule, decide the score.
The inventory itself needs an owner, distinct from any single model's owner
That role is accountable for the discovery process, the overdue-review count, and consistent rule application across rows, and is the person an examiner will ask to speak with first about the list as a whole.
Listen to it
The same lesson, talked through by two hosts as episode 33 of the podcast.
Real cases
Example 1: Earnest Operations LLC and the inventory the settlement effectively ordered (United States, consumer lending). Covered in depth in Sections 3A through 3C. The governance lesson in one line: a regulator's remedy, requiring a written corporate governance system of fair lending testing, internal controls, and risk assessments for AI models, describes, in substance, exactly the inventory and tiering artifact this topic teaches, imposed after the fact rather than built before harm occurred.
Example 2: The undocumented override as its own finding (United States, consumer lending). Also drawn from the Earnest investigation, and worth isolating from the bias finding because it is a distinct failure mode. Underwriters who bypassed the model without recording it left the organization unable to answer how often, and for whom, the automated decision was actually the decision made (Section 3G). An inventory row's validation status is only as meaningful as the organization's ability to say the model's stated behavior is its actual behavior, and an unlogged override breaks that chain quietly.
Example 3: SR 26-2's own narrowed model definition (United States, banking regulation, standing instrument). SR 26-2 excludes simple arithmetic calculations from its own definition of a model, a regulator explicitly declining to apply Tier 1 style scrutiny to Row 9's kind of system. (see Topic 4.6) for the deep evaluation-report treatment of metrics and evidence; the point here is narrower: even a banking supervisor's own rule distinguishes complexity levels, which is the second axis this topic's tiering rule scores directly.
Example 4: A well-tiered generative deployment (illustrative, cross-sector). An organization deploying a generative assistant for internal knowledge search scores it for materiality (moderate, an employee could act on a wrong internal policy summary) and complexity (high, generative and non-deterministic), lands it in Tier 1 despite having no external customer contact, and builds a monitoring plan that samples outputs for factual drift rather than assuming a one-time evaluation covers a system whose behavior can change with each update. Contrast this with a deployment that scores the same category of system as Tier 3 because "it's just internal," conflating audience with materiality, and misses exactly the generative-complexity signal Section 3H names.
Example 5: An independent challenge that starts from the inventory, not the report (illustrative, cross-sector, pointer to Topic 4.7). (see Topic 4.7) owns the deep treatment of independent challenge; the inventory connection is that a Tier 1 designation is what triggers the requirement for a second reader in the first place. An organization that skips the inventory step and lets each team decide informally whether its own model needs independent review will systematically under-challenge the models whose owners have the least appetite for scrutiny, which correlates poorly with which models actually carry the most risk.
Example 6: A conformity file built from a current inventory versus one built from memory (illustrative, cross-sector, pointer to Module 5). (see Topic 5.6) owns the deep treatment of the conformity file; here the point is that an organization assembling evidence for a high-risk system under the EU AI Act moves fastest when its inventory already names the system, its tier, its validation status, and its owner, because the conformity file is largely an extraction from an inventory that was already current, not a fresh research project undertaken under deadline pressure.
Example 7: The Epic Sepsis Model, read as a missing inventory row (United States, healthcare, pointer to Topic 4.6). (see Topic 4.6) owns the deep treatment of the evaluation-report failure in this case; read through this topic's lens, the deploying hospitals' deeper problem was that a system arrived bundled into their electronic health record and was never separately entered into a model inventory with an owner and a tier at all. It ran for years as software, not as a governed model, and that categorization, more than any single missing metric, is why nobody assigned it the validation depth its materiality (a life-safety decision aid) and complexity (a proprietary, opaque scoring model) should have triggered from day one.
Example 8: A merger's shadow-model discovery (illustrative, cross-sector). During due diligence for an acquisition, a buyer's governance team runs the discovery process from Section 3G against the target company and finds an AI-driven pricing tool embedded in a licensed vendor platform that never appeared on the target's own inventory, because procurement had classified the purchase as "software," not "a model." The buyer scores it under its own materiality-and-complexity rule, lands it at Tier 1 given its direct revenue impact and opaque logic, and requires it added to the combined inventory with an owner and a validation plan before close. The lesson generalizes past mergers: any organization's discovery process should periodically re-run the same procurement sweep on itself, not only when a transaction forces the question.
Example 9: A regulator's document request answered from a current inventory (illustrative, cross-sector). An examiner requests, on short notice, a list of every automated system involved in a specific business line's customer-facing decisions. An organization with a current, tiered inventory filters its existing rows by purpose and produces a defensible answer within a day, each row already carrying an owner, a tier, and a validation status the examiner can follow up on individually. An organization without one spends the same window reconstructing the list from memory, emails, and hurried interviews, arriving at an answer that is slower, less complete, and, because it was assembled under pressure specifically to satisfy the request, far less credible than a list that existed before anyone asked. The difference between the two organizations is not the quality of their models; it is whether the inventory discipline in Section 3I was already running.
Example 10: An enterprise risk register mistaken for a model inventory (illustrative, cross-sector). An organization's enterprise risk team maintains a single line item, "AI and automation risk," inside its broader risk register, reviewed annually alongside cyber risk, market risk, and operational risk more generally. When a new compliance hire asks to see the model inventory, this line item is what gets produced. It names no individual systems, no owners, no materiality or complexity scores, and no tiers, because it was built at the granularity an enterprise risk function normally works at, not at the per-model granularity Section 3B requires. The gap surfaces only when a regulator asks a question the line item cannot answer, such as which specific systems make automated credit decisions, and the organization discovers it has to build the real inventory from scratch under exactly the deadline pressure Example 9 warns against.
Example 11: A first-year inventory that surprised its own builders (illustrative, cross-sector). An organization building its first formal model inventory, following the Section 3N approach of starting with its highest-materiality function, expects to find perhaps a dozen systems across the business. The procurement sweep alone surfaces thirty-one AI-labeled features bundled into existing vendor contracts, most never separately reviewed because they arrived as part of a larger platform purchase. Rather than treating this as evidence the process went wrong, the inventory owner treats it as the process working exactly as intended: the honest count was always thirty-plus, and the organization simply had not looked until the discovery process from Section 3G gave it a reason to.
Where people go wrong
- "We validated our most important model, so we're covered." One strong evaluation report proves one system was checked. It says nothing about the systems the organization did not mention. Earnest's failure was not a single bad model; it was the inability to produce a complete, current account of what it ran at all. An inventory, not a report, is what an examiner asks for first.
- "A human reviews the output, so the materiality is low." Section 3D and the Section 5 scenario both name this directly. A human in the loop changes what kind of monitoring is appropriate; it does not by itself lower the harm a wrong output could still cause, especially at volume, where review quality degrades the same way clinician attention degrades under a flood of false alarms. (see Topic 4.6) Score materiality on the worst realistic outcome, not on who is nominally watching.
- "If it's not called a model, it doesn't belong on the model inventory." A spreadsheet with an embedded regression, a vendor feature bundled into a larger platform, and a pilot that quietly became production are all shadow models by function, regardless of what anyone calls them. The inventory tracks decisions and outputs, not job titles or department budgets.
- "Our generative assistant isn't covered by SR 26-2, so it doesn't need to be tiered." SR 26-2 expressly excludes generative and agentic AI from its own scope, but that is a statement about the reach of one regulation, not a statement about risk. Section 3H is explicit: the discipline transfers by practice even where the current regulatory authority does not yet require it, and waiting for a mandate is choosing exposure, not avoiding it.
- "An override is fine as long as a person made the call." An override without a record is the specific failure the Earnest investigation surfaced. The problem is not that humans overrode the model; it is that nobody could later say how often, for whom, or why, which makes the model's stated behavior and its actual behavior two different, unreconciled things.
- "Tiering everything the same way is fairer." Giving every model light review under-protects the systems that carry real consequence. Giving every model full Tier 1 review makes the inventory too expensive to maintain, which predicts decay: rows go unreviewed, and new systems get added off the books because doing it properly feels too costly. Tiering exists so Tier 1 rigor stays affordable by being reserved for the models that earn it.
- "The inventory is done once it's written." An inventory with no review-date discipline is a snapshot the organization will treat as current long after it stops being true. Overdue Tier 1 rows are systems running on expired evidence, the same failure this program names for evaluation reports with no review-and-revoke trigger. (see Topic 4.6)
- "Deleting a retired model's row keeps the list clean." Deletion erases the history an examiner, or an internal investigation, might need later, for instance to understand what logic governed a decision made two years ago. Mark a row retired, with a date and a reason, and keep it.
- "A vendor's bundled model doesn't need our own tiering, since the vendor already validated it." A vendor's validation is a claim about the vendor's testing, on the vendor's terms, the same population-transfer problem Topic 4.6 names for a vendor's marketed benchmark. (see Topic 4.6) Your inventory still scores materiality and complexity for how the system is used in your organization, and your validation status still states what you, not the vendor, independently confirmed.
- "Our EU AI Act risk classification already covers this." (see Topic 5.3) The Act's high-risk category and this topic's materiality-and-complexity tier answer different questions, one legal, one operational, and they can disagree. A system outside the Act's high-risk annexes can still be your organization's Tier 1 by this rule, and a system inside the Act's high-risk category still needs your own materiality and complexity scoring to decide its internal validation depth, because the Act tells you your legal obligations, not how much internal scrutiny is prudent.
- "A blended risk score is easier to read than two separate numbers." A single blended score hides exactly the reasoning a hostile reader will pull on: whether materiality was scored down because a human reviews the output, or whether complexity was scored down because a system is "mostly" deterministic. Two visible numbers, summed by a stated rule, let a reader check the reasoning; one blended number asks them to trust it.
- "Tiering is a one-time exercise at intake." A system's materiality and complexity are properties of what it does today, not what it was approved to do at launch. A model retrained, repurposed, or connected to a new decision workflow needs its scores revisited, the same version-reopening discipline Section 3I applies to validation status applies to the tier itself.
- "A high Tier 1 count means we did something wrong." Section 3L names this directly: the tier distribution is an output of the rule applied honestly, not a target with a preferred shape. An organization whose core function is lending, fraud detection, or another severe-consequence activity should expect a Tier 1 heavy inventory, and reshaping the distribution to look more comfortable is the identical error to under-scoring a single row to avoid its true tier.
- "The inventory is one team's job, so no single owner is needed for the whole thing." Without a distinct inventory owner, per Section 3M, different teams apply the tiering rule inconsistently, review dates slip in whichever team is busiest, and cross-functional systems that do not obviously belong to any one team are the most likely to fall through entirely. The inventory needs the same kind of single, named accountability the rows themselves require.
Questions people ask
- What is model inventory?
- A living register of every model and automated decision system an organization operates, with each row carrying a name and version, an owner, a purpose, a materiality score, a complexity score, a resulting tier, a validation status, and a next review date. The artifact an examiner asks for before any single system's evaluation report.
- What is materiality score?
- A 1 to 3 rating of what it costs the organization, a customer, or the public if a given model is wrong, scored on the worst realistic outcome rather than the average case, and never lowered solely because a human reviews the output.
- What is complexity score?
- A 1 to 3 rating of how traceable a model's reasoning is and how much its output can vary without a human choosing it to vary, from a fixed deterministic calculation (1) to an opaque, adaptive, or generative system (3).
- What is tier (Critical, Significant, Limited)?
- The three-level classification produced by summing the materiality and complexity scores (5 to 6, 3 to 4, or 2), which determines a model's required validation depth, revalidation cycle, and sign-off authority.
- What is shadow model?
- Any system making or materially shaping a decision that never entered the model inventory, commonly arriving through an unrecognized spreadsheet formula, a bundled vendor feature never separately reviewed, or a pilot that quietly became production.
Keep going
This lesson builds Model risk management and independent challenge, and that page shows the roles that hire for it. Every Certified AI Governance Professional (CAIGP) lesson.
Watch the lesson and prove it
Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.
Watch the lesson and prove it →