Skip to main content

What a model cannot know: hallucination produced on demand, then caught

The short answer

Hallucination is structural, not a bug

A language model generates the most plausible continuation of your prompt, not the most true one, and it has no built-in step that checks a claim against a source of record. Fabrication is a consequence of the mechanism that makes the model useful. You govern a system that invents by design; you do not wait for a version that stops.

What you will be able to do

  • Define hallucination precisely as fluent, confident output that is false or fabricated, and explain why it is a structural feature of how language models generate text, not a bug that a better model simply removes.
  • Distinguish the model's confidence from its correctness, and state why the tone of an answer carries no information about whether the answer is true.
  • Identify the specific gaps in what a model can know (facts past its training cutoff, private or unpublished facts, precise citations and numbers it was never given, anything requiring live retrieval it does not have) and predict where a given prompt will push the model into fabrication.
  • Induce a hallucination deliberately from a real AI tool available today, using at least three distinct provocation patterns, and capture the confident false output as evidence.
  • Catch each fabrication with a repeatable verification method that does not rely on the model policing itself, and explain why "ask the model if it is sure" fails.
  • Analyze a fabrication after the fact: name which knowledge gap produced it, which provocation triggered it, and which control would have caught it in production.
  • Produce a hallucination probe log: a documented set of prompts that induced confident falsehoods from your own model, each paired with the catch method that flagged it, ready to feed the "how my model fails" explanation you build next.

The ideas, one by one

Confidence is not correctness

A fabricated answer arrives in exactly the same fluent, specific, authoritative tone as a true one. The tone carries no information about truth. The features that make you trust an answer are the features a hallucination reproduces perfectly. This is the exact mistake that put nonexistent cases into a federal filing in the Michael Cohen matter in 2023.

You can predict fabrication by naming what the model cannot know

The five gaps (post-cutoff facts, private or unpublished facts, precise citations and numbers, live data it cannot retrieve, and its own certainty) tell you in advance where any prompt will push the model into invention.

The most specific claims are the most dangerous

Precise citations, exact numbers, verbatim quotes, and working links sit at the thin end of the model's knowledge, where fabrication is most likely and where a wrong answer does the most damage because it looks the most authoritative.

The model cannot police itself

Asking "are you sure?" produces the most plausible response to that question, often a fresh fabrication with a fresh fake source. Every real control brings in a source that is not the model.

Grounding is the strongest production control

Giving the model real source text and requiring it to answer only from that, with quotes, converts an easy-to-fabricate recall task into an easy-to-check reading task. It does not remove the need to check; it makes checking cheap.

Ordinary use, not attack, causes the worst incidents

Cohen was not attacking his tool; he was using it as he thought a search engine works. The most damaging fabrications come from innocent, everyday requests at scale, unsupervised. Governance has to assume the accidental trigger, not just the malicious one.

The check is the cheapest part of the incident

Independent verification of a specific, decision-driving claim takes minutes; skipping it can put a fabrication into a court record, a customer's inbox, or a board deck. Verification is not overhead; it is the control that stands between a draft and a headline.

A caught fabrication is a governance asset

The teardown (which gap, which provocation, which control) turns a single false answer into a durable fix and into documented evidence that you understood your own system's failure modes before you deployed it.

Name the hallucination type to reach the right control

Factual fabrication, citation fabrication, false attribution, instruction-unfaithfulness, and overconfident extrapolation are caught by different checks. "The tool hallucinated" is not actionable; "the tool fabricated a citation, caught by source-opening" is. Precision in naming the failure is precision in the fix.

The workflow, not the willpower, is the control

Teams skip verification under deadline pressure, the authority illusion, and habituation, not out of ignorance. Build the check into the process as a required, unskippable step so that doing the right thing is the path of least resistance, rather than trusting individuals to remember under load.

Mark your certainty honestly

That models fabricate and that grounding helps are established; that a given tool "rarely hallucinates now" is emerging and must be tested; that a model will police itself is speculative and must never be a deployed control. A recommendation that overstates certainty is its own fabrication.

Listen to it

The same lesson, talked through by two hosts as episode 9 of the podcast.

Real cases

One note before the cases. The point of collecting six is not to prove that AI fails, which needs no proof, but to show that the same small skeleton underlies fabrications that look entirely different on the surface: a legal filing, a support chat, a phone notification, a health tip, a court brief, a vendor claim. Different industries, different stakes, one mechanism. If you can run the teardown on these six, you can run it on the seventh you meet in your own work, whatever its costume.

These are real, documented cases. Each is analyzed with the three-question teardown from Section 3G so the pattern becomes portable. Michael Cohen and Google Bard is this topic's anchor and is treated in depth in the scenario and throughout; the others are referenced to show the same mechanism across domains, and each is owned in depth by another topic where noted.

Example 1: Michael Cohen's fake cases (Google Bard, 2023), the anchor. Michael Cohen used Google Bard to find legal cases supporting a motion to end his supervised release early. He believed Bard was, in his own words, a "super-charged search engine." It produced citations to cases that do not exist. His attorney David Schwartz included them in a filing in the Southern District of New York. Judge Jesse Furman went looking for the cited cases and found that none of them existed (NPR, 2023; the court's March 2024 order declined to sanction Cohen or Schwartz but noted the citations were nonexistent). Teardown: the gap was precise citations the model never stored (Gap 3), triggered by demanding specific supporting cases (Pattern 1), and independent source verification (Control 1) would have caught every one, but nobody ran it before filing. This is the whole topic in one incident: an ordinary research request, a confident fabrication, a skipped check, a court record.

Example 2: A support bot invents a company policy (2025). An AI customer-support agent for a software tool told users it had a policy restricting use to one device, a policy that did not exist. Users canceled subscriptions over the fabricated rule before the company clarified. Teardown: the gap was a private, internal fact the model was never given (Gap 2); the trigger was an ordinary user question about policy (Pattern 3, and entirely accidental); grounding the bot on the actual published policy (Control 2) would have prevented it. This case is owned in depth by Topic 1.6 as the "explain how your model fails to a skeptic" anchor. (see Topic 1.6)

Example 3: AI news summaries invent headlines (2025). A major consumer device maker's AI feature generated notification summaries that produced false headlines attributed to real news organizations, including a summary that misstated a real story. The feature was monitored and then disabled. Teardown: the gap was precise, faithful reproduction of specific source content the model compressed and distorted (a mix of Gaps 3 and 4); the trigger was ordinary automatic summarization at scale; the control is grounding plus faithfulness checking against the source article (Controls 2 and 3). Owned in depth by Topic 3.4 as the ship-and-rollback anchor. (see Topic 3.4)

Example 4: Health advice built on a fabrication (2025). A person following AI dietary advice replaced table salt with sodium bromide and was hospitalized with bromism, a form of poisoning. The model produced confident guidance in a domain where a specific wrong answer causes physical harm. Teardown: the gap was a safety-critical specific the model treated as ordinary text (Gap 3, applied to a domain where precision is life-safety); the trigger was an ordinary health question (Pattern 3); the control is domain gating and mandatory professional verification for medical claims (Controls 1 and 5). Owned in depth by Topic 5.2 as the AI-literacy anchor. (see Topic 5.2)

Example 5: A lawyer's brief attacked for hallucinated citations (2025). In litigation involving MyPillow, a federal judge sanctioned attorneys whose brief contained numerous AI-generated citations to cases that did not exist. Unlike the Cohen matter, here sanctions followed. Teardown: identical mechanism to the anchor (Gap 3, Pattern 1, Control 1 skipped), which is precisely why it recurs: the failure is structural and the control is easy to skip under deadline pressure. Owned in depth by Topic 11.1, where your own file comes under attack. (see Topic 11.1)

Example 6: A vendor's overstated accuracy metric (2024). A state attorney general settled with a healthcare AI vendor over how it represented its product's accuracy, including its "critical hallucination rate." The lesson here is one level up: not only do models hallucinate, but claims about how often they hallucinate can themselves be unverified. Teardown: this is why the eval suite in Module 4 must measure the fabrication rate you can reproduce yourself rather than trusting a vendor's number (Control 1 applied to the vendor's own claims). Owned in depth by Topic 4.2. (see Topic 4.2)

Lay the six side by side and the same skeleton appears in every one:

  • The gap is always nameable. Precise citations (Cohen, MyPillow), private facts (support bot), faithful reproduction of specifics (news summaries), safety-critical specifics (health advice), or a vendor's own unverified metric. Name the gap and you have the root cause.
  • The trigger is almost always ordinary use. A research question, a policy question, a routine summary, a health question. None of these are attacks. The damage comes from innocent requests at scale, which is why designing only against adversaries misses the real risk.
  • The output looked authoritative. In every case the fabrication carried the tone of a fact. Nobody was fooled by a hesitant answer; they were fooled by a confident one. This is the confidence-correctness decoupling doing its work in the field.
  • A cheap control existed and was skipped or never wired in. Open the citation, ground on the real policy, check the summary against the source, gate the medical claim, verify the vendor's number. In each case the control was known and inexpensive; the failure was that it was not required at the point of use.

That skeleton is portable. The next fabrication you meet, in any domain, has the same four bones: a nameable gap, an ordinary trigger, an authoritative surface, and a cheap control that was not enforced. The teardown from Section 3G is how you read it, and the workflow fix from Section 3I is how you close it for good.

One honest caveat about these cases. They are the ones that became public, which means they are the fabrications that were eventually caught and reported. The far larger number are the ones nobody noticed: the AI-drafted email with a slightly wrong figure that the customer never questioned, the summary that dropped a caveat and changed a meaning, the confident answer that shaped a decision and was never traced back. Public incidents are the visible tip; the discipline you are building exists to catch the invisible majority before they surface, in your own work, where no reporter is watching.

The through-line across all six: the mechanism is always a knowledge gap plus a plausible-continuation fill, the trigger is almost always ordinary use rather than attack, and the control that would have caught it is almost always a check against a source that is not the model, skipped under time pressure or never wired in.

Where people go wrong

  • "Hallucination is a bug that the next model version will fix." No. Fabrication is a structural consequence of how language models generate text: they optimize for plausible continuation, not truth, and they have no built-in step that checks a claim against a source of record. Newer models hallucinate less often on common topics and can hallucinate more convincingly on rare ones. You are governing a system that fabricates by design; the control is external verification, not a future patch.
  • "If the answer sounds confident and detailed, it is probably right." This is the exact reasoning that reaches court records. Confidence and detail are generated features that a fabrication reproduces perfectly. The tone of an answer carries no information about its truth. Specificity is if anything a warning sign, because the most specific claims sit at the thin end of the model's knowledge where fabrication is most likely.
  • "You can just ask the model whether it is sure." The model cannot accurately report its own certainty (Gap 5). Asking "are you sure?" produces the most plausible response to that question, which may be a confident reassurance or a reflexive apology followed by a fresh fabrication. Self-checking is not a control. Every real control brings in a source that is not the model.
  • "Only obscure or adversarial prompts cause hallucination." The most damaging fabrications come from ordinary, innocent requests: a normal research question, a routine policy lookup, a standard summarization. Michael Cohen was not attacking Bard; he was using it as he thought a search engine works. Ordinary use at scale, unsupervised, is where hallucination does its real damage.
  • "Retrieval (RAG) makes hallucination impossible." Grounding the model on real documents dramatically reduces fabrication and makes it far easier to catch, which is why it is the strongest production control. But a grounded model can still misread the source, over-reach beyond it, or stitch provided text into a claim the source does not support. Grounding converts an easy-to-fabricate recall task into an easy-to-check reading task; it does not remove the need to check.
  • "A hallucination is the same as a lie." A lie requires knowing the truth and choosing to state otherwise. The model has no access to truth and no intent; it generates plausible text with no idea whether it is true. This distinction matters for governance: you cannot fix fabrication by appealing to the model's honesty, because there is no honesty to appeal to. You fix it with controls that assume the output is unverified.
  • "If the model gives a source, the claim is verified." A model that fabricates a fact will fabricate a citation for it just as easily. The existence of a named source proves nothing. The source is worth something only after you open it and confirm it exists and says what the model claimed. Never let a citation substitute for opening the citation.
  • "Checking is too slow to be worth it in production." The check is the cheapest part of the entire incident. Judge Furman ran the check in an afternoon and found the fabrication; the cost of skipping it was a filing built on nonexistent cases under a real person's name. Verification of specific, decision-driving claims is not overhead; it is the control that stands between a draft and a headline.
  • "If two different models agree, the answer must be true." Agreement between models lowers the odds of a random fabrication, but it does not prove truth. Both models trained on largely overlapping public text can share the same common misconception, the same outdated fact, or the same popular myth, and both can be confidently wrong in the same way. Cross-model agreement is a useful early filter, not a verification. The specific, high-stakes claim still gets checked against a source of record, not against a second model's echo.
  • "Hallucination only matters for text; structured outputs are safe." A model asked for a structured output (a table, a JSON record, a list of citations, a set of figures) fabricates inside the structure just as readily, and the tidy formatting makes the fabrication look more authoritative and machine-verified than free text. A neat table of fake case citations is more dangerous, not less, because its form signals rigor. Structure is a presentation choice; it adds no truth. Verify the contents regardless of how clean the container looks.
  • "A disclaimer that the tool can make mistakes is a sufficient control." A generic disclaimer shifts liability language but changes no behavior at the moment of use, when a confident answer is about to be acted on. Users habituate to disclaimers and ignore them. A disclaimer is not a control; a control is a step in the workflow (grounding by default, mandatory source-opening, a required second reviewer for high-stakes claims) that actually intercepts the fabrication before it acts. Do not mistake a warning label for a safety mechanism.

Questions people ask

What is hallucination (in AI)?
Model output that is fluent and confident but false or fabricated: a statement, citation, quotation, number, or entity presented as fact that does not correspond to reality. It arises because the model generates the most plausible continuation of a prompt rather than a verified truth, and has no built-in step that checks a claim against a source of record.
What is confabulation?
An alternative term for hallucination, used in the United States National Institute of Standards and Technology Generative AI Profile (NIST AI 600-1, 2024) to name confidently stated but erroneous or fabricated content. Preferred by some researchers because "hallucination" borrows a clinical term for perceiving things that are not present, and NIST's own framing cautions that terms like "hallucination" risk implying the system has perceptions or intentions it does not have. Either word names the same behavior; neither implies the model is aware of what it is doing. More on Confabulation
What is plausible continuation?
The core mechanism of a language model. Given the text so far, the model generates the statistically most likely next piece of text based on patterns in its training data. It optimizes for plausibility, not truth, which is why fluent output can be confidently false.
What is confidence-correctness decoupling?
The fact that the confidence and specificity of a model's answer carry no information about whether the answer is true. A fabrication is delivered in the same authoritative tone as a fact, so tone cannot be used as a truth signal.
What is training cutoff?
The fixed date up to which a model's training data extends. The model has no knowledge of events, rulings, releases, or facts after this date and will often fabricate a plausible answer rather than admit the gap when asked about recent matters. More on Training cutoff

Keep going

Watch the lesson and prove it

Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.

Watch the lesson and prove it →