Skip to main content

Auditing your own data: provenance, gaps, and quiet poison

The short answer

Three distinct problems, three distinct fixes

Provenance (where it came from), gaps (what is missing), and quiet poison (what is silently wrong) are separate failures. Document provenance honestly, scope gaps to the system's real decisions, and hunt poison actively. Confusing them produces the wrong response.

What you will be able to do

  • Analyze the provenance of a dataset your organization uses: reconstruct where each part came from, when, from whom, and under what legal basis or license.
  • Distinguish the three failure classes of a dataset (provenance you cannot prove, gaps you did not notice, and poison you did not detect) and explain why each is a distinct governance problem with a distinct fix.
  • Trace a dataset back through the AI systems inventory you built earlier to the specific system it feeds, so the audit attaches to a real consequence, not an abstract file. (see Topic 0.2)
  • Detect the common forms of quiet poison: mislabeled licenses, undocumented scraped content, personal data collected without a basis, illegal content, poisoned or manipulated records, and content that has drifted since it was collected.
  • Judge whether a dataset is fit to keep using, must be quarantined, or must be rebuilt, and state the reasons in language that survives a regulator's or auditor's challenge.
  • Produce a data audit artifact for one real dataset: a provenance record, a gap list, a poison-risk register, and a clear keep, quarantine, or rebuild recommendation.
  • Explain why a dataset that references external links rather than storing content is a provenance problem in itself, using the LAION structure as the worked example.
  • Convert each audit finding into a named obligation with an owner, so the audit produces commitments the organization can be held to rather than observations it can forget.
  • Read a vendor's data documentation critically: for what it omits, for claims you can verify against primary sources, and for laundering language that states conclusions in place of evidence.

The ideas, one by one

Undocumented provenance is the precondition for every other failure

A dataset with no datasheet or record of origin is not merely under-documented; it is the exact condition in which mislabeled licenses, unconsented data, and illegal content hide. Start every audit by separating what you can prove from what you cannot, and mark the unprovable clearly.

Performance is not provenance

LAION-5B trained high-performing models for over a year while containing illegal content. Good metrics tell you the model fits its data, never that the data is legal, licensed, representative, or clean.

The hunt step can override everything

A single instance of illegal content forces removal regardless of accuracy, coverage, or business value. Fitness is not a weighted average; some findings are decisive on their own.

Scraped and link-based data are a higher-risk category by construction

Nobody selected each item, the content may be mutable, and the source cannot prove what it held at training time. Split-view poisoning has been shown practical and cheap [Carlini et al., 2023]. Record whether content was stored-and-hashed at ingestion or merely linked.

Stated licenses are unreliable more often than not

A 2024 audit found license omission over 70 percent and error over 50 percent on popular hosting sites. Verify against the original source, not the aggregator, and log unconfirmed licenses as open risk.

Gaps are present-tense, not future backlog

A coverage or temporal gap is a region where a deployed system is making confident, unearned decisions right now, about people or cases the data does not represent.

Merges and acquisitions are where provenance breaks

Two sources with different origins, licenses, and consent bases flattened into one table hide the break. The audit's job is to un-flatten them.

The audit ends in a judgment, not a description

Trace, map, hunt, then judge keep, quarantine, or rebuild, each with defensible reasoning that names the residual risk. A description of present data is one move of four.

This audit is the seed of your evidence trail

The artifact you produce here becomes the first entry in your data provenance file, cited later in your conformity file and evidence annex. Governance that survives an audit was documented from the first dataset, not scrambled together at the end. (see Topic 2.6) (see Topic 10.6)

Every finding must become an obligation

An audit that ends in observations has failed. Write the required action next to each finding in the same document, so the audit is a set of commitments the organization can be held to, not a set of concerns it can admire and forget.

One owner, a real cadence, a durable home

An audit run once by whoever had time is not a control. Name a single accountable owner per dataset, re-audit on change (new source, changed terms, expanded use) not just the calendar, and store the result in a versioned provenance file so the next owner inherits a record, not a mystery.

Detection is specific, not a glance

Each poison class has a concrete method: hash-matching (escalated) for illegal content, duplicate detection across splits, license tracing to the primary source, identifier scanning for personal data, ingestion hashing for link-based sources. "I looked and it seemed fine" is not a hunt.

Anonymization is a mitigation, not a clearance

Removing names can be reversible through re-identification and does nothing about the original consent or the source's provenance. Log it as one mitigation and keep resolving the consent and origin questions separately.

Read vendor documentation for what it omits

A vendor datasheet detailed on composition but silent on collection method and legal basis has told you where the risk is by omission. Verify the checkable claims against primary sources, and treat "ethically sourced" or "publicly available" as unproven conclusions, not clearances.

Fitness is decided per source, not per dataset

One dataset routinely yields three verdicts: keep the clean in-house source, quarantine the scraped one, restrict the licensed one to its permitted use. "The dataset is good" skips the analysis; the output is a decision per source with its own reason and obligation.

Listen to it

The same lesson, talked through by two hosts as episode 11 of the podcast.

Real cases

These examples show provenance, gap, and poison analysis applied to real, documented cases. Each is described accurately; none is used beyond what the public record supports.

Example 1: LAION-5B and the quiet poison in a scraped index (the anchor). LAION-5B was a dataset of 5.85 billion image-and-text pairs, assembled by scraping links from the open web, and used to train Stable Diffusion and other image generators. In December 2023 the Stanford Internet Observatory reported it contained thousands of links to suspected CSAM (3,226 suspected, 1,008 externally validated). LAION, the nonprofit behind it, took the datasets down and later released a cleaned Re-LAION-5B in 2024 with child-safety organizations. Read through the four moves: provenance was weak by construction (scraped links, no per-item origin, content mutable); the gap analysis is secondary here; the poison (illegal content) was decisive and forced removal. No amount of model performance would have justified keeping it. This is why the hunt step can override everything else.

Example 2: The Data Provenance Initiative license audit (mislabeled licensing at scale). The 2024 Nature Machine Intelligence audit of more than 1,800 text datasets found that the licenses recorded on popular hosting sites were omitted more than 70 percent of the time and wrong more than 50 percent of the time. The teaching point is not one bad dataset but a base rate: if your audit accepts stated licenses without verification, you are trusting a signal that is unreliable most of the time. The fix is to verify licenses against the original source, not the aggregator, and to log any license you could not confirm as an open risk.

Example 3: Split-view poisoning as a demonstrated, cheap attack (deliberate poison). Carlini and co-authors (2023) demonstrated that link-based web datasets can be poisoned by an attacker who buys expired domains appearing in the dataset, estimating poisoning of 0.01 percent of LAION-400M or COYO-700M for around 60 US dollars. This moves data poisoning from a theoretical worry to a documented, low-cost, practical attack. For the auditor, it means link-based datasets carry an adversarial risk that stored-and-hashed datasets do not, and the audit should record whether ingestion captured and hashed content or merely followed links.

Example 4: Datasheets for Datasets as the documentation instrument (the provenance fix). Gebru and co-authors (2018) proposed that every dataset ship with a datasheet answering standardized questions across motivation, composition, collection, processing, uses, distribution, and maintenance. This is the concrete artifact that turns "we should document our data" into a repeatable form. When your audit finds a dataset with no datasheet, the remedy is not vague; it is to author one, and the gaps you cannot fill in that datasheet are your provenance findings written down.

Example 5: The link-rot dimension of drift (drifted content). Research on dataset decay has repeatedly documented that URLs in web datasets stop resolving or change content over time (the general phenomenon known as link rot and content drift). For a link-based training set, this means the data a model trains on today is not the data the dataset's authors documented, even absent any malice. An auditor records, for any link-based source, the fraction of links still resolving and whether original content was preserved at ingestion.

Example 6: Acquisition merges as a provenance break (chain-of-custody failure). A recurring pattern in enterprise data (documented across data-governance practice rather than a single headline) is the acquisition merge: Company A buys Company B and folds B's customer data into A's systems. B collected that data under its own notices, consents, and licenses, which may not cover A's intended AI use. The merged table looks like one clean source. The audit's job is to detect that the single table has two origins with two legal bases, and to treat the portion whose basis does not cover the new use as a consent gap and a provenance break.

Example 7: The New York Times v. OpenAI as inherited provenance (upstream data you did not collect). In December 2023 The New York Times sued OpenAI and Microsoft, alleging its content was used to train models without permission. (see Topic 5.5) for the deep treatment. The audit lesson here is narrower and specific to provenance: when your system is built on a general-purpose model, the training-data decisions of that vendor become part of your system's provenance whether you inspected them or not. A downstream deployer's audit is incomplete if it stops at the data the deployer collected and ignores the data the foundation-model vendor collected on the deployer's behalf. The provenance question now extends upstream of your own organization.

Example 8: Re-LAION-5B as the shape of a real remediation (what fixing provenance costs). When LAION rebuilt its dataset as Re-LAION-5B in 2024, it did so in partnership with the Internet Watch Foundation, the Canadian Centre for Child Protection, and Stanford, matching the dataset's links against more than sixteen million hashes of known illegal content and removing 2,236 identified links. The teaching point is scale and specialization: cleaning quiet poison out of a web-scale set required dedicated child-safety organizations, hash databases the general public cannot access, and a coordinated rebuild. This is why the correct move for an individual auditor who suspects illegal content is to escalate to those who hold that capability, not to improvise a check. Remediation of serious poison is a specialist, cross-organization effort, not a solo cleanup.

Example 9: The Data Provenance Initiative's tooling response (auditing at scale). Beyond documenting the license problem, the same 2024 effort built tools and standards to trace dataset lineage (source, creators, licenses, and downstream use) across more than 1,800 datasets. The lesson for an organization is that provenance tracing is a discipline that scales with the right structure: the individual audit you run by hand is the same reasoning that, standardized and tooled, can cover a whole data estate. The four moves do not change with scale; only the tooling does.

Where people go wrong

  • "Our data is fine; the model performs well." Performance measures whether the model fits the data it has, not whether the data is legal, licensed, representative, or clean. LAION-5B trained high-performing models for over a year while containing illegal content. Good metrics are not a provenance audit and never substitute for one.
  • "We scraped it from the public web, so we can use it." Public accessibility is not a license, not consent, and not a guarantee of legality. Scraped data can carry copyright exposure (see Topic 5.5), personal data with no lawful basis (see Topic 2.2), and illegal content. "Scraped" is the start of the risk assessment, not the end of it.
  • "The dataset's license says we can use it, so we are covered." A 2024 audit found stated licenses on popular hosting sites were missing over 70 percent of the time and wrong over 50 percent of the time. Verify the license against the original source, not the aggregator, and log any you cannot confirm as an open risk.
  • "A gap is just missing data; we can add it later." A gap is a region where the model's confidence is unearned right now, in production, making decisions about people the data does not represent. Treating a coverage gap as a future backlog item ignores that the system is already acting inside the hole.
  • "Poison means someone attacked us deliberately." Most quiet poison is not an attack. It is mislabeled licenses, drifted content, duplicated splits, unconsented personal data, and illegal content that arrived through ordinary scraping and merging. Deliberate poisoning is real and cheap [Carlini et al., 2023], but it is one category among several.
  • "We stored a datasheet, so provenance is handled." A datasheet is only as good as its honesty. A datasheet that asserts a clean origin the authors could not actually verify is worse than none, because it launders unprovable provenance into apparent fact. The value is in recording what you cannot prove, not papering over it.
  • "Link-based datasets are equivalent to stored datasets." They are not. A dataset of links cannot prove what its content was at training time, is exposed to split-view poisoning, and decays through link rot. When you audit a link-based source, record whether content was captured and hashed at ingestion; if not, provenance is weaker by construction.
  • "The audit is done when I have described the data I have." The audit is done when you have also mapped what is missing and hunted for what is hidden, and reached a keep, quarantine, or rebuild judgment. Describing present data is one of four moves, not the whole job.
  • "Anonymized data has no provenance or consent problems." Removing obvious identifiers does not automatically make data non-personal, and it does not retroactively supply a lawful basis for the original collection. Re-identification is often possible by combining an anonymized set with other data, and the consent question about how the data was gathered in the first place does not disappear when names are stripped. Anonymization is a mitigation to record in the provenance, not a clearance that closes the audit. (see Topic 2.2)
  • "Once audited, always audited." A clean audit is true as of its date and its sources. The moment a new source is added, an existing source changes its content or terms, or the system is expanded to a higher-stakes use, the prior audit no longer covers the current reality. Treat the audit as a control with a cadence, not a certificate that never expires.
  • "Escalating illegal-content suspicion is overreacting; I should confirm it first." Attempting to confirm suspected illegal content yourself can mean unlawfully possessing or processing it and destroys the clean handling that legal and law-enforcement processes require. The correct action is to preserve, isolate, and escalate to those with the authority and tooling, not to open the files and look.
  • "The dataset is either fit or unfit." Fitness is decided source by source, not for the dataset as a whole. A single dataset routinely contains a clean in-house source you keep, a licensed source fit for one use but not another, and a scraped source you quarantine. A verdict of "the dataset is good" or "the dataset is bad" skips the analysis; the real output is a decision per source, each with its own reasoning and obligation.
  • "Finding a problem means the audit failed." An audit that surfaces a quarantine or a rebuild has succeeded, not failed. The failure is an audit that certifies clean what was never checked, exactly the state that preceded the LAION-5B discovery. A finding is the audit doing its job; the goal is an accurate, defensible picture of the data, not a clean bill of health.

Questions people ask

What is data provenance?
The documented history of a dataset, including its origin, the legal basis or license covering its collection and use, the chain of custody from origin to your system, the transformations applied along the way, and whether a written record (such as a datasheet) travels with it. Strong provenance can answer, for every meaningful part of a dataset, where it came from and under what permission. More on Data provenance
What is datasheet (for a dataset)?
A standardized written record documenting a dataset's motivation, composition, collection process, processing, recommended uses, distribution, and maintenance. Proposed by Gebru and co-authors in 2018 and modeled on the datasheets that accompany electronic components. The concrete instrument for recording provenance, including honestly recording what is unknown.
What is chain of custody?
The sequence of hops a dataset takes from its origin to your system (vendors, exports, merges, labeling, storage moves). Each hop is a point where the provenance record can break and where contamination can enter; the highest-risk hop is usually a merge or acquisition. More on Chain of custody
What is gap (data gap)?
The absence of data the system needed. Forms include coverage gaps (populations the system acts on but the data underrepresents), temporal gaps (missing or stale time ranges), edge-case gaps (rare consequential cases with too few examples), label gaps (missing or inconsistent labels across sources), and consent gaps (data held with no record of agreement).
What is quiet poison?
Content in a dataset that should not be there and that degrades, biases, or legally endangers the system trained on it, without announcing itself. Includes illegal content, mislabeled or missing licenses, personal data without a lawful basis, deliberately poisoned records, drifted content, and duplicated records across training and test splits.

Keep going

Watch the lesson and prove it

Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.

Watch the lesson and prove it →