Scaling Laws
Train a bigger language model on more data with more compute and the loss falls in a straight line on a log-log plot. This is the closest thing modern AI has to a law of physics, and almost everything about its long-run shape is contested. Kaplan and collaborators at OpenAI published the first careful fit in January 2020. Two years later DeepMind's Chinchilla paper showed Kaplan's recipe was wrong by roughly a factor of four on the data axis, and the industry quietly reorganized around the correction.
The case
Kaplan et al. (2020) trained transformer language models across seven orders of magnitude of compute and fit clean power laws. Test loss L declines as L(N) ∝ N^(-0.076) in parameters and L(D) ∝ D^(-0.095) in tokens, with a separate power law in compute C. Each curve held smoothly across model sizes from ~10³ to ~10⁹ parameters. The conclusion that drove the field for two years: given a fixed compute budget, parameters should grow roughly as C^0.73 and data only as C^0.27. Translation: spend most of your compute making models bigger, not feeding them more tokens.
Hoffmann et al. (2022) at DeepMind ran the experiment Kaplan did not. They fixed compute and swept the parameter-to-token ratio, training over 400 models. Their three independent estimation methods agreed: compute-optimal training requires parameters and tokens to scale equally, roughly N ∝ C^0.5 and D ∝ C^0.5. They then trained Chinchilla, a 70B-parameter model on 1.4 trillion tokens, and it beat Gopher (280B parameters, 300B tokens) on nearly every benchmark while using the same compute.
The implication was brutal for the field's existing models. GPT-3 (175B parameters, 300B tokens) was undertrained by roughly 4×. So was PaLM, Gopher, Jurassic, and most published 2020-2022 models. They were burning compute on parameters they could not feed.
Where it shows up
| Model | Year | Parameters | Training Tokens | Tokens / Parameter |
|---|---|---|---|---|
| GPT-3 | 2020 | 175B | 300B | 1.7 |
| Gopher | 2021 | 280B | 300B | 1.1 |
| Chinchilla | 2022 | 70B | 1.4T | 20 |
| Llama 2 | 2023 | 70B | 2T | 28 |
| Llama 3 | 2024 | 70B | 15T | 214 |
Notice the last row. Meta's Llama 3 trained on roughly 10× the Chinchilla-optimal token count for a 70B model. They were not trying to be compute-optimal at training. They were minimizing inference cost. A smaller, more thoroughly trained model is cheaper to serve forever after training ends. The compute-optimal frontier is the wrong frontier if you plan to run the model a trillion times.
Scaling laws also predicted, before they were observed, the emergent capabilities discussed in concept emergent capabilities. The loss curve is smooth; downstream task performance is not, but the underlying log-likelihood improvement that drives it is. This is what made the concept bitter lesson feel newly urgent in 2020: methods that scale clean keep winning.
What's contested
Three open empirical questions, each worth billions.
Does the curve bend? Kaplan and Chinchilla fits are clean within their tested range. Extrapolating four more orders of magnitude in compute, which is roughly where frontier labs project 2027-2030 models will sit, assumes the power law holds. There is no strong theoretical reason it must. Several groups (Caballero et al. 2022, "Broken Neural Scaling Laws") have argued the true curve has inflection points the early fits missed.
Are we running out of data? Villalobos et al. (2022, "Will We Run Out of Data?") estimated high-quality public text would be exhausted between 2024 and 2027 at current consumption rates. If Chinchilla-optimal training requires D ∝ C^0.5, and compute is growing ~4× per year, data has to grow ~2× per year forever. The internet does not. Synthetic data, multimodal data, and reinforcement learning on verifiable rewards are the three live hypotheses for what replaces text.
Does loss reduction translate to capability past a point? A 0.1 nat drop in loss bought GPT-2 → GPT-3 a qualitative leap. Whether the next 0.1 nat buys another leap, or diminishing returns set in around human-level next-token prediction, is unknown. The scaling laws fit loss, not usefulness.
Why this has to do with other realms
The scaling laws are an empirical regularity of the kind physics rarely produces outside thermodynamics: a clean power law spanning many orders of magnitude, with no first-principles derivation. Compare to the concept drake equation from astrobiology, where each factor is independently estimated and the product is mostly noise. Scaling laws are the opposite shape: the macro curve is sharp, but the micro mechanism (why exactly does loss fall as N^(-0.076) and not N^(-0.1)?) is unexplained. Theoretical physics has spent a century looking for laws this clean. ML found one by accident in eighteen months of GPU time.
There is also a concept bitter lesson parallel to the history of computing: Moore's Law was also an empirical regularity with no theoretical derivation, and the industry organized capital allocation around it for fifty years before it bent.
An open question
If the loss curve holds for four more orders of magnitude of compute, what does the resulting model not know how to do? The honest answer in 2026 is that no one has produced a defensible list. That gap, between predicted loss and predicted capability, is where the next decade's argument will sit.
Key sources
- Kaplan et al., "Scaling Laws for Neural Language Models" (arXiv:2001.08361, January 2020) — the original fit, the paper that reshaped pretraining budgets industry-wide.
- Hoffmann et al., "Training Compute-Optimal Large Language Models" (arXiv:2203.15556, March 2022) — the Chinchilla correction, with the experimental design Kaplan didn't run.
- Caballero et al., "Broken Neural Scaling Laws" (arXiv:2210.14891, 2022) — the case that the clean power law hides inflection points.
- Villalobos et al., "Will We Run Out of Data?" (arXiv:2211.04325, 2022) — the data-exhaustion projection.
- to verify: the Llama 3 technical report's discussion of training past compute-optimal for inference economics (Meta AI, 2024).
Further reading
- The Bitter Lesson by Rich Sutton (2019 essay, incompleteideas.net) — the philosophical precursor that scaling laws made quantitative.
- concept emergent capabilities — what the smooth loss curve produces on downstream tasks that does not look smooth at all.
- Dario Amodei, "Machines of Loving Grace" (2024 essay) — what an Anthropic founder thinks the curve buys, written by someone who has seen the internal training runs.
- Epoch AI's compute and dataset trend dashboards (epochai.org) — the empirical tracker for whether the curve is still holding.
Abhishek's take
I pick the smaller model for range notes when it gives the buyer the same answer on a vendor sheet and returns before the line review starts. Scaling laws matter to me less as prophecy than as a buying constraint: the cheapest useful answer, repeated all season, beats the grander answer nobody can afford to run.
See Also
- concept emergent capabilities (the downstream phenomenon scaling laws underwrite)
- concept bitter lesson (the prior the scaling laws confirmed)
- concept transformer architecture (what is being scaled)
- concept compute overhang (the strategic situation scaling laws create)
- concept drake equation (cross-realm: another famous power-law-of-our-ignorance)
- concept moores law (cross-realm: empirical regularity organized an industry, then bent)