Model Collapse
A model trained on its own descendants does not merely repeat itself. In Shumailov et al. 2023, rare events vanish first, because each generation samples a thinner version of the original data. The average sentence survives. The odd fact, minority pattern, and low-frequency edge case disappear.
How it works
Model collapse is a recursion problem. A model learns a distribution from human data, generates new samples, then a later model trains on a mixture that includes those samples. If this repeats, the synthetic part can push the training distribution toward high-probability outputs and away from the tails.
The small equation is the whole trap:
D_next = (1 - a)D_real + aD_model
a is the synthetic-data share. When a rises across generations, the model is no longer learning the world directly. It is learning a sampled memory of a sampled memory.
Shumailov et al. call this “the curse of recursion” in their 2023 arXiv paper. Their core result is not that all generated data is poison. It is that unmarked, recursively reused generated data can erase information that was rare in the original distribution.
Where it shows up
The risk is clearest in domains where the tail matters.
| Domain | What gets lost first | Why it matters |
|---|---|---|
| Language | rare facts, dialects, small archives | retrieval starts sounding confident but narrow |
| Images | unusual poses, defects, local styles | generation drifts toward median compositions |
| Code | obscure APIs, old bug fixes | assistants miss the case outside common tutorials |
| Science text | negative results, failed experiments | literature search inherits publication bias twice |
A web full of model text is not automatically doomed. The danger comes when provenance is missing. A 2026 crawler that cannot tell lab notes from generated summaries has a weaker map than a 2016 crawler that only had messy human pages.
What is contested
The open question is dose. Shumailov et al. 2023 and Alemohammad et al. 2023 show failure modes under recursive training, but production training mixes are not public enough to measure the real synthetic share by source, language, and topic. assumption: the tail risk is higher in low-resource languages and niche technical domains because the original human corpus is smaller.
There is also a useful counterpoint. Synthetic data can help when it is filtered, labeled, and used for targeted gaps. The problem is not “model-made text exists.” The problem is model-made text entering the bloodstream as if it were fresh observation.
Why this belongs outside AI
Model collapse is the AI version of an old measurement problem: once the instrument changes the thing being measured, the next measurement is less independent. That makes it closer to concept fermi paradox than it first looks. Both problems ask whether absence is evidence or just a damaged sensor.
The space pages give the better instinct. mission voyager 1 matters because its signal is tagged, dated, directional, and physically constrained. dest proxima centauri reminds me that distance is not an opinion. mission breakthrough starshot lives or dies on whether its numbers can survive contact with engineering. AI training data needs the same discipline: provenance, date, source class, and a refusal to treat every token as equal.
Open question
What would a public “data provenance layer” for the web look like if it had to survive 10 model generations without becoming another spam signal?
Key Sources
- Shumailov et al. 2023, “The Curse of Recursion: Training on Generated Data Makes Models Forget,” arXiv:2305.17493 — the load-bearing model-collapse paper.
- Alemohammad et al. 2023, “Self-Consuming Generative Models Go MAD,” arXiv:2307.01850 — parallel evidence from recursive synthetic training.
- Shannon 1948, “A Mathematical Theory of Communication,” Bell System Technical Journal — the older grammar for thinking about signal, noise, and lost information.
Further Reading
- Shumailov et al. 2024, Nature, “AI models collapse when trained on recursively generated data” — peer-reviewed version of the recursion result.
- “Self-Consuming Generative Models Go MAD” — useful because it studies the failure as a feedback loop, not only a language-model issue.
- concept fermi paradox — a clean bridge from bad observations to bad conclusions.
Abhishek's take
I see the same failure on the floor when last season’s buy sheet becomes the quiet teacher. If the tools I wrote ingest only approved range notes, they start making safe repeats and bury the odd fabric swatch that a buyer kept because one store sold through it. I now treat a vendor photo and a dated fit comment as separate evidence, not decoration.
See Also
- concept fermi paradox
- mission voyager 1
- dest proxima centauri
- mission breakthrough starshot