Transformer Architecture — How Attention Changed Everything

A transformer does not read a sentence the way a human mouth speaks it: during training, every token position can be processed at once, and each position scores every other position. That one move broke the sequential bottleneck that limited recurrent neural networks. Since Vaswani et al. published “Attention Is All You Need” in 2017, the same basic design has powered BERT, GPT-3, AlphaFold 2-style protein models, code models, image generators, and speech systems. The price is brutal: full attention over 100,000 tokens means 10 billion pairwise scores per layer.

How attention works

Each token is turned into three learned vectors: query, key, and value.

Query: what this position is looking for.
Key: what this position can be matched by.
Value: what this position contributes if selected.

The core computation is:

softmax(QKᵀ / √d) V

The dot product QKᵀ scores which tokens matter to which other tokens. The softmax turns those scores into weights. The value vectors carry the information forward.

This separation matters. The word “bank” can be queried as a financial object in “deposit at the bank” and as a landscape object in “sit on the river bank.” The model does not store one meaning for the word. It builds a context-shaped representation from the other tokens nearby.

Multi-head attention repeats this process in parallel. The original 2017 transformer base model used 8 attention heads; the larger version used 16. Different heads can learn different routing patterns: previous-token copying, bracket matching, name binding, long-range reference, or phrase completion. The heads are not hand-coded. Gradient descent finds them if the training task rewards next-token prediction.

Why it beat recurrent models

Recurrent neural networks process tokens one step at a time. Token 1 influences token 100 only after passing through 99 intermediate updates. Long-range information has to survive a narrow hidden state.

A transformer lets token 1 and token 100 interact in one attention step. That makes training easier to parallelize on GPUs and TPUs, where dense matrix multiplication is the native food. The win was not only conceptual. It was hardware-shaped.

The catch is the square. If the sequence length doubles, attention work grows by 4×. A 2,048-token context has about 4.2 million token-pair scores per layer. A 100,000-token context has 10 billion. This is why long-context models are not free memory machines; they are expensive search systems with a learned index.

What has been found inside

Mechanistic interpretability has made the most progress on small circuits, not whole models. The cleanest example is the induction head, described by Anthropic researchers in 2022.

Given a pattern like:

A B C ... A B ?

an induction circuit helps predict C. One head copies information about previous tokens. A later head uses that copied information to find an earlier matching sequence and retrieve what came next. This is a primitive form of in-context learning: the model adapts to a pattern in the prompt without changing its weights.

That does not mean the whole model is understood. A 175-billion-parameter model like GPT-3 contains too many interacting components for circuit-level explanation to cover the system. The honest state is narrower: researchers have mapped some gears, not the engine.

What's contested

“Emergent abilities” are the loudest disputed claim. Wei et al. argued in 2022 that some large-model abilities appear suddenly at scale. Schaeffer, Miranda, and Koyejo argued in 2023 that many such jumps may be measurement artifacts caused by thresholded metrics rather than true phase transitions.

Both claims can be partly true. Scaling laws from Kaplan et al. 2020 and Hoffmann et al. 2022 show smooth loss improvements with more compute, data, and parameters. Human-visible tasks can still look sudden when a smooth internal improvement crosses a benchmark’s pass line. The open question is whether reasoning itself has thresholds, or whether our tests make gradual gains look like cliffs.

Why this has to do with other realms

Transformers are a physics story disguised as software. Attention is a dense interaction graph: every token can exert influence on every other token, like an all-to-all coupling system rather than a chain. That makes concept emergence more than a metaphor, because macro-behavior can appear from many small learned interactions that no single head “knows.”

They also raise a philosophy problem. A language model can produce a step-by-step explanation after the computation that produced the answer. Humans do this too. The link to concept free will is not that models have agency; it is that both humans and machines can generate persuasive post-hoc stories about hidden processes.

An open question

If attention is the expensive part and in-context learning is the prized part, can a cheaper architecture keep the second while cutting the first?

Key Sources

Vaswani et al., “Attention Is All You Need” (2017) — the original transformer paper.
Olsson et al., “In-context Learning and Induction Heads” (2022) — the key mechanistic interpretability reference for induction heads.
Kaplan et al., “Scaling Laws for Neural Language Models” (2020) — early quantitative scaling law work from OpenAI.
Hoffmann et al., “Training Compute-Optimal Large Language Models” (2022) — the Chinchilla paper on data versus parameter tradeoffs.
Wei et al., “Emergent Abilities of Large Language Models” (2022) — the main paper arguing for sudden capability jumps.
Schaeffer, Miranda, and Koyejo, “Are Emergent Abilities of Large Language Models a Mirage?” (2023) — the strongest counterclaim on benchmark artifacts.

Abhishek