Deduplication
Removing exact or near-duplicate documents from a training or fine-tuning corpus before training, the highest-leverage, lowest-cost mitigation against extractable memorization, since repetition frequency is the strongest predictor of what a model will memorize verbatim.
Defined in 2 GAGE programs, which carry 3 distinct definitions of it. The wording above is taught in AI Data Governance: The Data Chair.
How each discipline defines it
The same term does different work depending on who is using it. These are the definitions as each program teaches them, unedited.
Removing exact or near-duplicate documents from a training or fine-tuning corpus before training, the highest-leverage, lowest-cost mitigation against extractable memorization, since repetition frequency is the strongest predictor of what a model will memorize verbatim.
The process of identifying and merging duplicate records in a dataset; small business contact lists commonly contain a meaningful share of duplicates (an industry rule of thumb is roughly 15% to 25%).
Logic, typically keyed to an idempotency key or a similar unique identifier, that recognizes a repeated attempt at the same operation and discards or merges it rather than processing it as a new, separate event.
Where it is taught
The exact lessons this term appears in. The first 7 topics of every program are free with a free account.
- Data Readiness for Small Operations · Digital Foundations, AI Literacy & Professional Conduct
- Membership inference and extraction: what a model reveals about its training data · Poison, Leaks, and the Adversary, AI Data Governance: The Data Chair
- Pipeline forensics: reconstructing what the data did when the logs disagree · Data Incidents, AI Data Governance: The Data Chair
Terms it appears with
Not an alphabetical neighbourhood: these are the terms taught in the same lessons, ranked by how often they appear together.