Deduplication
Removing exact or near-duplicate documents from a training or fine-tuning corpus before training, the highest-leverage, lowest-cost mitigation against extractable memorization, since repetition frequency is the strongest predictor of what a model will memorize verbatim.
Defined in 2 GAGE programs, which carry 3 distinct definitions of it. The wording above is taught in Certified AI Data Governance Professional (CADGP).
How each discipline defines it
The same term does different work depending on who is using it. These are the definitions as each program teaches them, unedited.
Removing exact or near-duplicate documents from a training or fine-tuning corpus before training, the highest-leverage, lowest-cost mitigation against extractable memorization, since repetition frequency is the strongest predictor of what a model will memorize verbatim.
The process of identifying and merging duplicate records in a dataset; small business contact lists commonly contain a meaningful share of duplicates (an industry rule of thumb is roughly 15% to 25%).
Logic, typically keyed to an idempotency key or a similar unique identifier, that recognizes a repeated attempt at the same operation and discards or merges it rather than processing it as a new, separate event.
Where it is taught
The exact lessons this term appears in. The first module of every program is free with a free account.
- Data Readiness for Small Operations · Digital Foundations, Certified AI Practitioner: Workplace Foundations
- Membership inference and extraction: what a model reveals about its training data · Poison, Leaks, and the Adversary, Certified AI Data Governance Professional (CADGP)
- Pipeline forensics: reconstructing what the data did when the logs disagree · Data Incidents, Certified AI Data Governance Professional (CADGP)
Where it shows up at work
A definition is where Deduplication starts. The lessons above teach it inside the competencies below, which are what job descriptions ask for and what the GAGE credential records. Every link here is drawn from the same careers map that builds a role's readiness check.
Understands prompt injection, data poisoning, model theft, insecure integrations and excessive agent privileges, and the controls that reduce each.
One lesson on this page teach it as a primary outcome.
Finds unapproved AI use, sets which tools are approved and what may be pasted, and detects leakage without policing every keystroke.
One lesson on this page teach it as a primary outcome.
Roles that hire for it
Each role page shows the full competency set, what the seat decides, and a readiness check against your own evidence.
- Cybersecurity Risk Analystneeds working knowledge
- Chief Information Security Officer, AI security focusneeds it at depth
- Chief Technology Officer, AI engineering editionneeds it at depth
- AI Incident Response Leadneeds it at depth · 1 open posting this week
- AI Security Architectneeds it at depth · 8 open postings this week
- AI Governance Engineerneeds it at depth · 7 open postings this week
How job descriptions say it
Postings rarely use the textbook word. These are the phrases the careers map reads as asking for the competencies above.
Where the frameworks place it: OWASP Top 10 for LLM Applications; Google SAIF; NIST AI RMF MEASURE 2; ISO/IEC 27001 Annex A.
One interview question about it
What are the security failure modes specific to AI systems, and which conventional control covers none of them?
Asked of candidates for ai security fundamentals. Answering it in your own words is exactly how a GAGE topic is passed, and the credential records the pass where an employer can check it.
If you come from another field
Some backgrounds already carry this competency, and the readiness check gives credit for it.
- Cybersecurity and IT carries it as a claim to back up: Threat modeling and control design transfer; the AI-specific threats are new.
- A computer science or software engineering degree carries it as a claim to back up: Security fundamentals are a course you took, not yet a control you ran.
- A cybersecurity or information assurance degree carries it as work you can show: Security fundamentals are your coursework and your labs.
Terms it appears with
Not an alphabetical neighbourhood: these are the terms taught in the same lessons, ranked by how often they appear together.