Zipf's Law

Take any large English text. Count word frequencies. Rank them from most common down. Plot frequency against rank on a log-log scale. You will get an almost-straight line with slope close to −1. The, of, and, to, a, in, that… each is about half as common as the one before it. This holds for Mandarin, Sanskrit, ancient Greek, Tagalog, Yorùbá, Bengali. It holds for languages that share no family. It even holds, approximately, for the undeciphered concept indus valley script and the concept linear a tablets, which is one of the better arguments that those scripts encode language at all.

That is Zipf's law. Its formal statement: frequency of the n-th-ranked item scales as ≈ 1 / n^α, where α is close to 1.

George Kingsley Zipf, a Harvard linguist, popularised the regularity in 1949 in Human Behavior and the Principle of Least Effort, though it had been observed by stenographer Jean-Baptiste Estoup in 1916 and by Edward Condon in 1928. Zipf claimed it as a universal cost-minimisation principle: speakers minimise effort by reusing a small vocabulary; listeners minimise effort by demanding precision; the equilibrium is the 1/n distribution. Modern linguistics does not fully buy this explanation, but the empirical pattern has held up for 75 years.

At a glance

Each rank's frequency is roughly the previous rank's frequency divided by n/(n-1). The same shape holds for city populations, surname counts, web traffic, corporate revenues, and species abundance in many ecosystems.

Where else it appears

Zipf-like distributions show up in:

City populations. New York, Los Angeles, Chicago, Houston, Phoenix… US cities are close to 1/n in population. Same for India, Japan, Germany. France is famously a poor fit (Paris is too dominant), and some emerging economies depart sharply, which makes the pattern a diagnostic.
Surname frequencies. Smith, Johnson, Williams, Brown, Jones in English; Zhang, Wang, Li, Zhao, Liu in Chinese; Patel, Singh, Kumar, Sharma in India. The exponent varies but the shape is consistent.
Web-page traffic. Site visits, file downloads, video plays. This is why the long-tail business model works: there is a long tail, predictably distributed.
Corporate revenues. Fortune 500 firms ranked by revenue trace a Zipfian distribution roughly within sector. The bigger sector-wide picture is a different power law (Pareto, see below).
Species abundance. In many ecological surveys the n-th most abundant species is about 1/n as numerous as the most abundant. The fit is imperfect (Preston's lognormal is often better) but Zipf is the first-order approximation.
Earthquake magnitudes (Gutenberg–Richter law) and wars by casualty count (Lewis Fry Richardson, 1948) follow related power laws though with different exponents.

The pattern is so common that the right question is not "why this distribution?" but "why this specific exponent?"

Candidate mechanisms

There is no single accepted derivation. Several mechanisms produce Zipf-like distributions, and reality probably uses more than one:

Preferential attachment (Yule 1925, Simon 1955, Barabási–Albert 1999). New events attach to existing categories in proportion to their current size. Already-popular words get reused; already-large cities attract more migrants. The Yule–Simon process generates a Zipfian tail.
Cost-minimisation (Zipf 1949, Mandelbrot 1953). Mandelbrot showed that a communication system minimising joint speaker-listener cost reproduces approximately Zipf's exponent. This is the rigorous version of Zipf's hand-wavy "least effort" argument.
Random text models (Mandelbrot 1953, Miller 1957). If a monkey types random letters including spaces, the resulting "word" frequencies are Zipfian, even with no language structure. This embarrassed early linguists who thought Zipf was a deep claim about meaning; it remains a warning that Zipf-distributed data does not by itself imply anything cognitive.
Critical phenomena (Bak, Tang & Wiesenfeld 1987). concept power laws arise generically at the critical point of a phase transition. If language, cities and ecosystems all sit at or near self-organised criticality, Zipf falls out for free.

The Mandelbrot–Miller objection is important: Zipf's law is necessary but not sufficient evidence of underlying structure. You can get the shape with no information content at all.

A useful cousin: the Pareto distribution

What economists call the Pareto distribution (top 20% of population hold 80% of wealth, etc.) and what linguists call Zipf's law are mathematically equivalent statements of the same underlying power-law continuous distribution. Pareto (1896) ranked by value; Zipf (1949) ranked by frequency. They sit on the same curve.

Income, wealth and corporate revenues are typically described in Pareto language; words and cities in Zipf language; the literature stays separate by convention.

Why this has to do with other realms

The persistence of Zipf-like patterns across language, economics and ecology is one of the strongest arguments that there are general statistical laws of human and biological systems, distinct from the specific mechanisms of each domain. This connects to concept emergence, concept power laws and concept soc civilizations.

In cryptography it has an offensive use: the concept one time pad notwithstanding, most weak ciphers leak Zipfian frequency signatures of the underlying language. Frequency analysis is, at root, an attack on Zipf-distorted statistics.

In retail and merchandising it predicts the long tail: a small number of items dominate sales, but the tail is long enough that platforms which can stock infinitely cheaply (Amazon, Spotify, Netflix) win disproportionately. Long-tail strategy is operationalised Zipf.

An open question

Why does the exponent stay so close to 1 across systems that share no obvious mechanism? Mandelbrot's optimisation argument gives an exponent of 1 for specific (and questionable) cost assumptions. Preferential attachment gives 1 only at specific parameter values. The empirical exponent being roughly 1 in language, cities, surnames and species seems to demand a deeper unifying explanation we do not yet have.

Key sources

Zipf, G. K. (1949), Human Behavior and the Principle of Least Effort.
Mandelbrot, B. (1953), "An informational theory of the statistical structure of language."
Newman, M. E. J. (2005), Contemporary Physics, "Power laws, Pareto distributions and Zipf's law" — the modern survey.
Gabaix, X. (2009), Annual Review of Economics, "Power laws in economics and finance."

Abhishek's take

I see the same curve in the range: a few styles pay the rent, then a long tail keeps the customer from feeling boxed in. The trap is treating the tail as waste. On a 100-day lead time, one bad bet in the head hurts more than ten quiet experiments in the tail.

Abhishek