Grokking

In September 2022, Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin and Vedant Misra at OpenAI published a short paper with a striking plot. Train a small transformer on the algorithmic task of computing a * b mod p for a fixed prime p. Memorise the answers in the training set; achieve 100% training accuracy after a few thousand steps. Test accuracy stays at random levels (about 1/p).

Keep training. Training accuracy stays at 100%, test accuracy stays at chance, for tens of thousands of additional steps. The network is overfitting; the standard textbook answer is to stop training and accept the result.

Then, somewhere around step 50,000 to 100,000 — long after any reasonable early-stopping criterion would have triggered — test accuracy abruptly jumps to 100%. The network has not just memorised the table. It has generalised to the underlying algorithm. The transition happens fast: a few thousand steps from chance to perfect.

They named this grokking, after the Heinleinian sense of understanding something to its core.

The phenomenon is robust. It appears in modular arithmetic, symbolic-regression toy problems, permutation tasks, and at varying scales of model and dataset. It is the most legible currently known empirical example of delayed generalisation: a system that memorises first and understands later, with a sharp gap between the two states.

At a glance

Top line: train accuracy. Bottom line: test accuracy. The network memorises within a few thousand steps and then sits at chance test accuracy for tens of thousands more steps. Eventually the test accuracy snaps to 100%. The transition is fast once it starts.

What it looks like

The training curves are stereotyped:

Phase 1 (memorisation). Train loss falls to near zero quickly. Test loss stays high.
Phase 2 (plateau). Train accuracy at 100%, test accuracy at chance. Visually, the system looks done overfitting and not improving. Most practitioners would stop training.
Phase 3 (transition). Test loss drops abruptly over a comparatively short span of training steps. The transition is sharp on a linear axis and very sharp on a log axis.
Phase 4 (generalisation). Train and test accuracy both at 100% indefinitely.

The duration of Phase 2 (the plateau) is sensitive to weight decay, learning rate, and dataset size. Strong weight decay shortens the plateau dramatically; very small datasets prevent generalisation altogether.

What the network is doing

A naïve hypothesis: the network is memorising in Phase 1 and replacing memorisation with the right algorithm in Phase 3. Mechanistic interpretability work since 2022 has shown the picture is more subtle.

Nanda, Chan, Lieberum, Smith & Steinhardt (2023, ICLR) reverse-engineered a small transformer trained on a + b mod p and found a specific circuit:

The model represents each input number as a vector that lives on a discrete Fourier basis (with period p).
The MLP layer computes products of these Fourier components, producing what is effectively the trigonometric-identity expansion of the modular sum.
The output head reads off the relevant Fourier coefficient.

The clean Fourier-basis representation is present in the weights long before generalisation appears in the loss. What changes during grokking is not the discovery of the circuit but its amplification: the algorithmic circuit becomes large enough relative to the memorisation circuit to dominate the output.

In other words: the network is computing both the algorithm and the lookup table from very early on. The transition is when the algorithmic path gets cleaner and stronger than the memorisation path. Weight decay is what pushes this: it shrinks low-utility weights faster than high-utility ones, and the memorisation circuit has lower utility per parameter than the algorithmic one.

This picture is now well-established for modular arithmetic. Whether the same mechanism explains grokking-like phenomena in larger, less algorithmic tasks remains open.

Why this matters beyond toys

Three reasons grokking is studied harder than its small scale would suggest:

It is empirical evidence that "overfit" and "understanding" can both be present simultaneously, with the model picking one or the other based on training dynamics. The traditional dichotomy (overfitting versus generalisation) is too coarse.
It is a measurable phase transition. Most deep-learning phenomena are continuous and noisy. Grokking has a sharp transition that can be located in time and correlated with specific circuit changes. This makes it a rare clean experimental handle on what neural networks are actually doing.
It suggests delayed generalisation may be common at larger scales but hidden by less clean tasks. Recent work (Liu, Michaud, Tegmark 2023 and others) has argued that the smoother improvement curves of large language models may hide grokking-like dynamics at the circuit level. If so, training that looks like a slow climb may actually be many small grokking events stacking.

Open questions

Several specific puzzles, which the field has not closed:

Can grokking be predicted? Given a task and an initialisation, can we say in advance whether and when grokking will occur? The honest answer is no, despite several useful necessary conditions.
Is grokking related to the "lottery ticket" hypothesis (Frankle & Carbin 2018)? The two phenomena both concern the discovery of sparse useful subnetworks inside dense ones. Whether they are different views of the same mechanism is contested.
What is the role of the loss landscape? Recent work suggests grokking corresponds to the optimiser escaping a "memorising plateau" basin and finding a "generalising" basin elsewhere. The geometric picture is suggestive but not yet rigorous.
Does grokking-like behaviour explain in-context learning? Large language models learn to perform new tasks from a few examples in their context window without weight updates. This looks like algorithmic generalisation. Whether it is mechanistically related to the grokking observed during gradient training is an active research question.

Why this has to do with other realms

Grokking is one of the cleaner connections between deep learning and the broader literature on phase transitions in concept power laws and concept soc civilizations. The training dynamics look very similar to first-order phase transitions in physics, including the sharpness of the transition and the role of "order parameters" (the relative weight of memorisation vs algorithmic circuits) in driving it.

It connects to concept spaced repetition indirectly: human learning sometimes shows a similar shape, where a concept seems opaque for weeks and then crystallises overnight. The mechanism is presumably different, but the macroscopic curve is the same.

It connects to concept cellular automata in spirit: complex emergent behaviour from simple training rules, with sharp qualitative transitions at parameter thresholds.

And it connects to concept emergence philosophically. Grokking is one of the few cases where a clear high-level behaviour (generalisation) emerges from a low-level system (gradient descent on a loss) in a way we can trace nearly all the way through. That makes it a useful test case for what "emergence" means operationally.

An open question

If grokking is a phase transition driven by the relative strength of algorithmic and memorising circuits, and weight decay is what causes the transition, then the choice of optimiser is the choice of which kind of generalisation can emerge. What other phenomena are we accidentally suppressing or selecting by our default optimiser choices in industrial-scale training runs? The question is too large to answer now but is probably worth large-scale empirical investigation.

Key sources

Power, Burda, Edwards, Babuschkin & Misra (2022), "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets" — the original observation.
Nanda, Chan, Lieberum, Smith & Steinhardt (2023), "Progress Measures for Grokking via Mechanistic Interpretability" — the Fourier circuit reverse-engineering.
Liu, Michaud & Tegmark (2023), "Omnigrok: Grokking Beyond Algorithmic Data" — extension to non-algorithmic tasks.
Notsawo et al. (2024), latest survey — to verify reference details.

Abhishek's take

I see the useful warning here in our size-curve models: a model can look finished on last season's SKUs and still fail the first Monday a new drop lands. I keep a shadow scorecard for a 100-day lead-time buy, because the question is not whether the training file is clean, it is whether the model changes a sleeve-length call before the purchase order closes. The textbook version says stop when the curve is flat; the floor version says wait until the decision it changes is named.

Abhishek