Synthetic data: when it saves you and when it launders a bias
The short answer
Synthetic data is a tool with two opposite faces
It genuinely saves you when it protects real people (privacy surrogates) or stress-tests a system (edge cases). It launders a bias when it is asked to teach a model about real people using nothing but a reflection of a biased source. The same technique, pointed at different jobs, produces a fix or a cover-up.
What you will be able to do
- Define synthetic data and name its three practical families (rule-based augmentation, generative-model output, and simulation), and state what each can and cannot add to a dataset.
- Distinguish the cases where synthetic data genuinely helps (privacy protection, rare-event coverage, edge-case testing) from the cases where it only relabels a bias you already had.
- Explain the fidelity ceiling: a generator can only reproduce structure that was already in its source, so synthetic data cannot create information that the real data never contained.
- Analyze a proposed synthetic-data use for bias laundering by testing whether it adds real information or merely rebalances counts, using held-out real data from the group at issue.
- Explain model collapse (Shumailov et al., Nature, 2024) and why recursive training on synthetic output erases the tails of a distribution first, which are usually the minority and edge cases.
- Produce a synthetic-data decision record: a written go, no-go, or conditional-go decision with the test evidence, the safeguards, and the residual risk, ready to enter your data provenance file.
- Position the decision against the relevant rules: General Data Protection Regulation (GDPR) data-minimization, the fact that synthetic data is not automatically anonymous, and the EU AI Act's data-governance duties for high-risk systems (Article 10).
The ideas, one by one
The fidelity ceiling is the master idea
A generator or augmentation method can only reproduce structure already present in its source. It cannot add information the real data never contained. So generating "more" data for an underrepresented group does not fill the gap with truth; it fills it with the model's best guess, built entirely from the biased picture you were trying to escape.
Laundering has a precise signature
Aggregate and synthetic metrics improve while performance on the held-out real minority slice stays flat or drops. Equal counts are not equal information. The only honest test is measured on real, untouched data from the exact group at issue, never on synthetic validation data.
Model collapse eats the tails first (Shumailov et al., Nature, 2024)
Recursive training on synthetic output narrows a distribution generation by generation, and the rare cases vanish before the common ones. In real governance datasets the tail is usually the minority, so collapse is a machine for making the underrepresented disappear while average metrics still look fine.
Simulation is the one family that can add information
Augmentation and generative models are mirrors of your data. Simulation is a source built from an external model of the world, so it can add knowledge the source lacked, bounded by the accuracy of its assumptions. Do not expect a mirror to do a source's job.
Synthetic is not automatically anonymous
A generator can memorize real outliers, and synthetic records can sometimes be re-identified. Under the GDPR, "we used synthetic data" is not a privacy defense until you have tested for leakage and recorded it. Regulators (for example, the UK ICO) treat synthetic data as promising but conditional.
The EU AI Act expects a documented bias examination, not a silent fix
For high-risk systems, Article 10 contemplates synthetic or anonymized data for detecting and correcting bias under safeguards, and it expects the examination and the measures to be documented. An undocumented synthetic "balance" that does not help the real group is exactly what that duty is designed to catch. Treat the operational detail as emerging (stand-alone high-risk obligations deferred to 2 December 2027).
Recursion can reach you through your sources, not just your choices
As generated text and images fill the public web, data you scrape or buy may already be model output. That places you inside a collapse loop before you generate anything. "We do not use synthetic data" is only true if you have verified the provenance of what you ingest, which is why source provenance is a precondition, not an afterthought.
The decision, not the dataset, is the deliverable
A green fairness dashboard is what laundering produces, so it is not evidence. Your synthetic-data decision record, with the held-out-real numbers, the safeguards, and the residual risk routed to its owner, is the thing that survives a board and feeds your provenance file. Refusing a fake fix and recording why is stronger governance than shipping a documented illusion of fairness.
Listen to it
The same lesson, talked through by two hosts as episode 14 of the podcast.
Real cases
These are real, documented cases and results. Each is used to show one facet of the save-or-launder judgment. Jurisdictions and settings are varied deliberately.
Example 1: Model collapse in Nature (the anchor). Shumailov, Shumaylov, Zhao, Papernot, Anderson, and Gal, "AI models collapse when trained on recursively generated data," Nature, 2024. Training generative models recursively on their own output degrades them over generations; the diversity of output falls and the tails of the distribution vanish first. Demonstrated across large language models, variational autoencoders, and Gaussian mixture models, with a theoretical account of why it generalizes.
The governance lesson: a pipeline that feeds on synthetic data erases its rarest cases, which are usually the minority and the edge, while average metrics can still look acceptable. A later Nature author correction (2025) refined specifics without overturning the result.
Example 2: Fairness feedback loops amplify bias. Wyllie, Shumailov, and Papernot, "Fairness Feedback Loops: Training on Synthetic Data Amplifies Bias," ACM Conference on Fairness, Accountability, and Transparency, 2024. When models train on data earlier models generated, the resulting model-induced distribution shift disproportionately harms minoritized groups and can create representational disparity between groups even when the starting data was unbiased.
The lesson: synthetic loops do not merely preserve bias, they can manufacture it, so a dataset that started fair can end unfair after a round of synthetic augmentation. This is the rigorous form of laundering.
Example 3: SMOTE, the honest baseline. Chawla, Bowyer, Hall, and Kegelmeyer, "SMOTE: Synthetic Minority Over-sampling Technique," Journal of Artificial Intelligence Research, 2002. The foundational, transparent augmentation method: it creates new minority-class rows by interpolating between nearby real minority rows.
It is genuinely useful when a real minority pattern exists but is sparse, and it is honest about its ceiling: it cannot invent an example unlike anything in the real minority set. It is the reference point for "amplifying real but sparse signal," and the case that shows augmentation helps only when the underlying minority data is sound.
Example 4: The privacy-utility-fairness trade in differential privacy. A recurring finding across the differentially private synthetic-data literature (for example, work presented in the ACM FAccT and machine-learning venues, 2022 to 2025) is that adding the noise required for strong privacy guarantees costs accuracy, and that this cost is often not shared equally across groups: the smallest groups, with the least data, lose the most utility.
The lesson: even the strongest, most principled privacy technique can shift a burden onto exactly the minority you were trying to protect, so "private synthetic data" still needs a per-group check.
Example 5: Synthetic augmentation for health-data bias, with mixed results. A 2024 evaluation of synthetic data augmentation for mitigating covariate bias in health data was published in an Elsevier journal and indexed in PubMed Central. It found that synthetic augmentation helped in some configurations and did not in others, depending on the structure of the bias.
The lesson for an analyst: synthetic augmentation is not a reliable, one-size fix even in a careful academic setting; results are conditional, which is precisely why you test on held-out real data for your own case rather than trusting a general claim.
Example 6: Synthetic data as a privacy surrogate that must still be tested. Data-protection guidance from the UK Information Commissioner's Office and analogous bodies treats synthetic data as a promising privacy-enhancing technique while warning that it is not automatically anonymous and can permit re-identification if a generator memorizes outliers. The lesson: the strongest legitimate use of synthetic data (sharing without exposing real people) is only legitimate once you have tested the synthetic output for leakage and recorded the test.
Example 7: Generative models memorize and reproduce real training examples. Nicholas Carlini and colleagues, "Extracting Training Data from Diffusion Models" (USENIX Security Symposium, 2023), showed that a widely used image generator could be prompted to reproduce near-copies of specific real images from its training set, including identifiable photographs of people. This is direct evidence for the memorization risk behind the "synthetic is not automatically anonymous" rule: a generator does not always abstract away from its data; sometimes it stores and regurgitates a real individual. The lesson: before you rely on a generative model as a privacy surrogate, you must probe it for memorized real examples, because a synthetic dataset that reproduces a real person has leaked, not protected.
Example 8: The self-consuming loop is not hypothetical on the open internet. Multiple 2024 to 2025 analyses observe that as generated text and images fill the public web, models trained on freshly scraped data increasingly train on prior model output without anyone choosing to, which is the recursive condition that model collapse describes. The lesson for a governance analyst: you may be inside a synthetic feedback loop through your data sources even if your team never deliberately generated a single training row, so "we do not use synthetic data" is not a safe assumption unless you have verified the provenance of what you scrape or buy.
Read across the eight, the pattern is consistent. Synthetic data earns its place when it protects people or stress-tests a system and when its specific use has been measured against real data.
It launders a harm when it is asked to teach a model about real people using nothing but a reflection of a biased source.
Where people go wrong
- "We balanced the classes with synthetic data, so the bias is fixed." This is the central error. Equal counts are not equal information. If the synthetic minority records descend from a biased or thin real source, you have relabeled the bias, not removed it. The fairness metric improves because you engineered it to; the real-world harm to the real minority group is untouched or worse. Always measure on held-out real data from the group at issue.
- "Synthetic data is anonymous, so privacy is handled." Synthetic data is not automatically anonymous. A generator can memorize and reproduce a real outlier, and synthetic records can sometimes be re-identified or linked back to real individuals. The UK Information Commissioner's Office and other regulators treat synthetic data as a promising privacy technique that still requires a leakage test. "It is synthetic" is not a legal conclusion; it is a claim you have to prove.
- "More synthetic data is always better." Beyond a point, more synthetic data trained recursively causes model collapse (Shumailov et al., Nature, 2024): output diversity falls and the rare cases vanish. Volume is not the goal. A large pile of low-diversity synthetic records descended from a thin sample is worse than a small amount of real data, because it hides its own thinness behind sheer count.
- "The generator learned our data perfectly, so its output is as good as real." Learning your data perfectly means learning your data's flaws perfectly. A perfect mirror of a biased, incomplete dataset is a perfectly biased, incomplete dataset. Fidelity to the source is not the same as truth about the world. Only simulation from an external model of the world can add information the source lacked, and only to the extent the model is accurate.
- "Synthetic and real data are interchangeable in the training set." They are not, and the difference is the tails. Real data carries the rare combinations that keep a model honest about edge cases and minorities. Synthetic data, especially recursively generated, systematically thins those tails. Keeping enough real data in the mix is the documented antidote to collapse; a pure synthetic diet is degenerative.
- "We tested the synthetic data and it looked great." Tested against what? If you validated synthetic training data against synthetic validation data, you measured the copy against the copy. The only test that matters is performance on held-out real data for the specific group you care about. Self-consistency is not evidence.
- "Simulation and generative synthetic data are the same thing." They fail differently. Generative and augmentation methods are mirrors of your data and cannot add outside information. Simulation is a source built from an external model of the world and can add information, bounded by the accuracy of its assumptions. Conflating them leads people to expect a generative model to add knowledge it structurally cannot.
- "If our data is already messy with scraped web content, one more synthetic pass will not matter." If your source already contains model-generated content of unknown origin (scraped text, images, vendor data), you may already be inside a collapse loop before you generate anything. Adding a synthetic pass on top compounds it. This is why provenance of your source data (the next topic) is a precondition for any synthetic-data decision. (see Topic 2.5)
- "Regulators will accept synthetic data as a bias fix because the EU AI Act mentions it." The EU AI Act (Article 10) contemplates synthetic or anonymized data for detecting and correcting bias in high-risk systems, under safeguards, but that is permission to test carefully, not a blanket blessing to launder. The obligation is to document the examination for bias and the measures taken. An undocumented synthetic "fix" that does not improve the real group is exactly what Article 10 is written to catch.
- "Privacy noise protects everyone equally, so it is fair." The noise that differential-privacy techniques add to make synthetic data safe costs accuracy, and that cost is usually not shared evenly. In most configurations the smallest groups, with the least data, lose the most utility, because the same absolute noise is a larger share of a small group's signal. This is common but not universal: depending on the specific mechanism and the group's structure, a large but internally diverse group can sometimes absorb a proportionally bigger hit. The rule that survives either direction: never assume the noise lands evenly. A privacy claim needs a per-group utility check, not just an aggregate one.
- "Once we have the synthetic dataset, we can delete the real data." The real data holds the tail, the rare and minority cases that no generator can reconstruct once they are gone. Deleting real data to keep only synthetic data throws away the one thing that prevents collapse and the only honest measuring stick for whether the synthetic data works. Keep a governed real sample; do not trade it for storage savings.
Questions people ask
- What is synthetic data?
- Data that was manufactured by a process rather than measured directly from a real person, transaction, or event. It is designed to resemble real records statistically. It comes in three practical families: rule-based augmentation, generative-model output, and simulation. More on Synthetic data
- What is rule-based augmentation?
- Making more data by transforming real data with fixed rules, such as flipping an image, adding noise, or interpolating between nearby real minority rows (SMOTE). It can only recombine what is already present in the real data.
- What is SMOTE (Synthetic Minority Over-sampling Technique)?
- A 2002 augmentation method (Chawla et al.) that creates new minority-class records by interpolating between real minority records that lie near each other. Helpful when a real minority pattern is sound but sparse; harmful when the real minority data is itself skewed, because it amplifies the skew.
- What is generative-model synthetic data?
- Records produced by a model (a generative adversarial network, a variational autoencoder, a diffusion model, or a large language model) trained on real data to reproduce its distribution. Powerful and opaque; it reproduces the source's flaws as faithfully as its facts and smooths away rare cases.
- What is simulation?
- Data generated from a first-principles model of a process (a physics engine, a driving simulator, a fraud simulator). The one synthetic family that can add information the real data lacked, because its knowledge comes from the world model rather than from copying the dataset; only as accurate as its assumptions.
Keep going
This lesson builds Explainability, transparency and contestability, and that page shows the roles that hire for it. Every Certified AI Governance Professional (CAIGP) lesson.
Watch the lesson and prove it
Proving it happens inside the program: watch, explain it in your own words, and it goes on a record employers can check.
Watch the lesson and prove it →