Model collapse

Mechanism

Using synthetic data as training data can lead to issues with the quality and reliability of the trained model. Model collapse occurs for three main reasons:

1. functional approximation errors
2. sampling errors
3. learning errors

Importantly, it happens in even the simplest of models, where not all of the error sources are present. In more complex models the errors often compound, leading to faster collapse.

Disagreement over real-world impact

Some researchers and commentators on model collapse warn that the phenomenon could fundamentally threaten future generative AI development: As AI-generated data is shared on the Internet, it will inevitably end up in future training datasets, which are often crawled from the Internet. If training on AI slop (large quantities of unlabeled synthetic data) inevitably leads to model collapse, this could therefore pose a difficult problem.

Other researchers have disagreed with this argument, showing that if synthetic data accumulates alongside human-generated data, model collapse is avoided. The researchers argue that data accumulating over time is a more realistic description of reality than deleting all existing data every year, and that the real-world impact of model collapse may not be as catastrophic as feared.

An alternative branch of the literature investigates the use of machine learning detectors and watermarking to identify model generated data and filter it out.

Impact on large language models

In the context of large language models, research found that training an LLM on the synthetic data from a previous model causes a consistent decrease in the lexical, syntactic, and semantic diversity of the model outputs through successive iterations, notably for tasks demanding high levels of creativity.

In one study from 2024, a model was trained on its own output for nine generations. Afterward, it was given an input about the architecture of church towers. In response, the model produced nonsensical output including "the world's largest populations of black @-@ tailed jackrabbits, white @-@ tailed jackrabbits, blue @-@ tailed jackrabbits, ...".

Impact on ChatGPT

Around April 2026, ChatGPT output began frequently mentioning goblins, often enough to attract the attention of users. An OpenAI blog post described an over 3,881% increase depending on settings. This was only partially mitigated by a change to the system prompt, and third-party commentators described the output as a symptom of model collapse.

Diversity of model outputs

As models retrain on outputs sampled disproportionately from the higher-probability center of the distribution, rare words and uncommon syntactic constructions are among the first features to disappear. Statistical analysis of recursive next-token prediction training has shown that, when language models are trained recursively on synthetic data, the learned conditional distributions concentrate probability mass on a small subset of highly predictable continuations (a phenomenon characterized as "total collapse").

Alignment-related diversity reduction

Separately, output diversity reduction in LLMs can be attributed to post-training fine-tuning procedures rather than to recursive training on synthetic data. Reinforcement learning from human feedback (RLHF) and related preference-optimisation methods, which fine-tune models toward human-preferred responses, have been found to narrow output diversity at the level of generated text. A controlled analysis of the RLHF pipeline found that, relative to supervised fine-tuning, RLHF improved generalisation to out-of-distribution inputs but significantly reduced output diversity, a relationship described as a generalisation–diversity trade-off. The effect has also been observed at the conceptual level: when model outputs are treated as samples from a simulated population, aligned models exhibit lower conceptual diversity than their non-aligned counterparts, although no model studied reached the level of diversity found in human populations. Convergence toward a narrow set of high-reward responses under reinforcement-learning-based fine-tuning has sometimes been characterized as a form of mode collapse; this is related to, but mechanistically distinct from, the recursive form of model collapse described above, since it does not require synthetic data in the training set.

Measurement

Dedicated metrics have been developed to quantify reductions in output diversity. The Vendi Score, which draws on concepts from ecology and information theory, estimates the effective number of distinct elements in a sample without requiring a reference dataset. It has been applied to characterize model collapse, revealing that generative models reproducing every labelled mode of a dataset can nonetheless be substantially less diverse than the original data. For language specifically, diversity has been decomposed into separately measurable dimensions (lexical, syntactic, and semantic) with dedicated metrics developed for recursive-training experiments. Other instruments operate at higher levels of structure, such as the Sui Generis score, which measures the uniqueness of plot elements to evaluate diversity in narrative generation.

Mitigation

Proposed responses to diversity loss fall into several categories. Data filtering and watermarking, discussed in the context of model collapse more broadly above, address the training-data composition problem by identifying and excluding model-generated text before it re-enters training corpora. A further approach modifies the fine-tuning objective itself: because the standard cross-entropy objective maximises the likelihood of observed outputs without preserving alternatives, methods incorporating entropy regularisation have been proposed to discourage over-memorisation and maintain output variety while preserving task performance. Preserving output diversity in this way has also been reported to improve performance under repeated sampling at inference time.

Broader implications

Some researchers connect the reduction of output diversity to wider concerns about cultural and epistemic homogenisation. Studies of human–AI collaboration have reported that generative assistance can raise the quality of individual contributions while reducing the collective diversity of the resulting body of work, and analyses of large-scale story generation have found that model outputs frequently recombine a limited repertoire of plot elements. Such patterns have been linked to the prospect of a narrowing of the publicly available range of ideas, sometimes termed knowledge collapse. It has been argued that maintaining diversity across an ecosystem of distinct models, rather than relying on a small number of similar systems, may help counteract this tendency, though only up to a point.

Mathematical models of the phenomenon

1D Gaussian model

In 2024, a first attempt has been made at illustrating collapse for the simplest possible model — a single dimensional normal distribution fit using unbiased estimators of mean and variance, computed on samples from the previous generation.

To make this more precise, we say that original data follows a normal distribution ${\displaystyle X^0\sim \mathcal{N}(\mu ,{\sigma }^2)}$, and we possess ${\displaystyle M_0}$ samples ${\displaystyle X_j^0}$ for ${\displaystyle j\in \{1,\dots ,M_0\}}$. Denoting a general sample ${\displaystyle X_j^i}$ as sample ${\displaystyle j\in \{1,\dots ,M_i\}}$ at generation ${\displaystyle i}$, then the next generation model is estimated using the sample mean and variance:

${\displaystyle {\mu }_{i+1}=\frac{1}{M_i}\sum _j{X}_j^i;{\sigma }_{i+1}^2=\frac{1}{M_i-1}\sum _j(X_j^i-{\mu }_{i+1}{)}^2.}$

Leading to a conditionally normal next generation model ${\displaystyle X_j^{i+1}|{\mu }_{i+1},{\sigma }_{i+1}\sim \mathcal{N}({\mu }_{i+1},{\sigma }_{i+1}^2)}$. In theory, this is enough to calculate the full distribution of ${\displaystyle X_j^i}$. However, even after the first generation, the full distribution is no longer normal: It follows a variance-gamma distribution.

To continue the analysis, instead of writing the probability density function at each generation, it is possible to explicitly construct them in terms of independent random variables using Cochran's theorem. To be precise, ${\displaystyle {\mu }_1}$ and ${\displaystyle {\sigma }_1}$are independent, with ${\displaystyle {\mu }_1\sim \mathcal{N}\left(\mu ,\frac{{\sigma }^2}{M_0}\right)}$ and ${\displaystyle (M_0-1){\sigma }_1^2\sim {\sigma }^2\mathrm{\Gamma }\left(\frac{M_0-1}{2},\frac{1}{2}\right)}$, following a Gamma distribution. Denoting with ${\displaystyle Z}$ Gaussian random variables distributed according to ${\displaystyle \mathcal{N}(0,1)}$ and with ${\displaystyle S^i}$ random variables distributed with ${\displaystyle \frac{1}{M_{i-1}-1}\mathrm{\Gamma }\left(\frac{M_{i-1}-1}{2},\frac{1}{2}\right)}$, it turns out to be possible to write samples at each generation as

$X_j^0=\mu +\sigma Z_j^0,$

$X_j^1=\mu +\frac{\sigma }{\sqrt{M_0}}{Z}^1+\sigma \sqrt{S^1}{Z}_j^1,$

and more generally

$${\displaystyle X_j^n=\mu +\frac{\sigma }{\sqrt{M_0}}{Z}^1+\frac{\sigma }{\sqrt{M_1}}\sqrt{S^1}{Z}^2+\cdots +\frac{\sigma }{\sqrt{M_{n-1}}}\sqrt{S^1\times \cdots \times S^{n-1}}{Z}^n+\sigma \sqrt{S^1\times \cdots \times S^n}{Z}_j^n.}$$

Note, that these are not joint distributions, as ${\displaystyle Z^n}$ and ${\displaystyle S^n}$ depend directly on ${\displaystyle Z_j^{n-1}}$, but when considering ${\displaystyle X_j^n}$ on its own the formula above provides all the information about the full distribution.

To analyse the model collapse, we can first calculate variance and mean of samples at generation ${\displaystyle n}$. This would tell us what kind of distributions we expect to arrive at after ${\displaystyle n}$ generations. It is possible to find its exact value in closed form, but the mean and variance of the square root of gamma distribution are expressed in terms of gamma functions, making the result quite clunky. Following, it is possible to expand all results to second order in each of ${\displaystyle 1/M_i}$, assuming each sample size to be large. It is then possible to show that

${\displaystyle \frac{1}{{\sigma }^2}\mathrm{Var}(X_j^n)=\frac{1}{M_0}+\frac{1}{M_1}+\cdots +\frac{1}{M_{n-1}}+1+\mathcal{O}\left(M_i^{-2}\right).}$

And if all sample sizes ${\displaystyle M_i=M}$ are constant, this diverges linearly as ${\displaystyle n\to \mathrm{\infty }}$:

${\displaystyle \mathrm{Var}(X_j^n)={\sigma }^2\left(1+\frac{n}{M}\right);\mathbb{E}(X_j^n)=\mu .}$

This is the same scaling as for a single dimensional Gaussian random walk. However, divergence of the variance of ${\displaystyle X_j^n}$ does not directly provide any information about the corresponding estimates of ${\displaystyle {\mu }_{n+1}}$ and ${\displaystyle {\sigma }_{n+1}}$, particularly how different they are from the original ${\displaystyle \mu }$ and ${\displaystyle \sigma }$. It turns out to be possible to calculate the distance between the true distribution and the approximated distribution at step ${\displaystyle n+1}$, using the Wasserstein-2 distance (which is also sometimes referred to as risk):

${\displaystyle \mathbb{E}\left[{\mathbb{W}}_2^2\left(\mathcal{N}(\mu ,{\sigma }^2),\mathcal{N}({\mu }_{n+1},{\sigma }_{n+1}^2)\right)\right]=\frac{3}{2}{\sigma }^2\left(\frac{1}{M_0}+\frac{1}{M_1}+\cdots +\frac{1}{M_n}\right)+\mathcal{O}\left(M_i^{-2}\right),}$

${\displaystyle \mathrm{Var}\left[{\mathbb{W}}_2^2\left(\mathcal{N}(\mu ,{\sigma }^2),\mathcal{N}({\mu }_{n+1},{\sigma }_{n+1}^2)\right)\right]=\frac{1}{2}{\sigma }^4\left(\frac{3}{M_0^2}+\frac{3}{M_1^2}+\cdots +\frac{3}{M_n^2}+\sum _{i\ne j}\frac{4}{M_i{M}_j}\right)+\mathcal{O}\left(M_i^{-3}\right).}$

This directly shows why model collapse occurs in this simple model. Due to errors from re-sampling the approximated distribution, each generation ends up corresponding to a new step in a random walk of model parameters. For a constant sample size at each generation, the average distance from the starting point diverges, and in order for the end distribution approximation to be accurate, or for the distance to be finite, the sampling rate ${\displaystyle M_i}$ needs to increase superlinearly, i.e. one needs to collect increasingly more samples over time, perhaps quadratically. However, even in that case the expected distance after ${\displaystyle n}$ steps remains non-zero and the only case in which it does in fact end up being zero is when sampling is infinite at each step. Overall, this only shows us how far on average one ends up from the original distribution, but the process can only "terminate", if the estimated variance at a certain generation becomes small enough, effectively turning the distribution into a delta function. This is shown to occur for a general gaussian model in the subsection below. Empirical investigation has confirmed this theoretical analysis.

N-D Gaussian model

Furthermore, in the case of multidimensional model with fully synthetic data, exact collapse can be shown.

Linear regression

In the case of a linear regression model, scaling laws and bounds on learning can be obtained.

Statistical language model

In the case of a linear softmax classifier for next token prediction, exact bounds on learning with even a partially synthetic dataset can be obtained.