Dark data
A data source (an old warehouse table, a cached export, a spreadsheet a departed employee built) that feeds a pipeline without ever appearing in a formal source inventory or lineage map, because no one thought to register it. A chapter drafted only from named, registered sources will miss dark data by construction; the defensible response is a stated discovery method and date, not a claim that no dark data exists.
Defined in 2 GAGE programs, which carry 3 distinct definitions of it. The wording above is taught in AI Data Governance: The Data Chair.
How each discipline defines it
The same term does different work depending on who is using it. These are the definitions as each program teaches them, unedited.
Data an organization collects, generates, or stores that is never actively used, analyzed, or, most relevantly for this topic, monitored by anyone with clear accountability for it. Industry estimates of its share of total enterprise data vary widely by study and methodology; several studies over the past several years have suggested it commonly makes up a large share, in some cases a majority, of what organizations store, though any single percentage should be treated as illustrative rather than a fixed benchmark. A first-pass estate survey typically undercounts it because un-owned data is disproportionately data nobody is looking at.
Data that exists somewhere in an organization but is not on any current inventory, such as a forgotten export, an old shared-drive file, or a copy piped into an unapproved third-party tool. It cannot be scheduled for destruction until it is found, which is why an audit is a floor, not a ceiling.
A data source (an old warehouse table, a cached export, a spreadsheet a departed employee built) that feeds a pipeline without ever appearing in a formal source inventory or lineage map, because no one thought to register it. A chapter drafted only from named, registered sources will miss dark data by construction; the defensible response is a stated discovery method and date, not a claim that no dark data exists.
Where it is taught
The exact lessons this term appears in. The first 7 topics of every program are free with a free account.
- The un-owned dataset: finding data nobody claims and deciding who owns it now · The Estate Survey, AI Data Governance: The Data Chair
- Data governance for the conformity file: your chapter in the document the regulator reads · Lineage Under Audit, AI Data Governance: The Data Chair
- The retention decision: what you must keep, what you must destroy, and proving both · Data Reality, AI Governance: Applied Mastery
Terms it appears with
Not an alphabetical neighbourhood: these are the terms taught in the same lessons, ranked by how often they appear together.