Bayesian Inference
A test that is 99% accurate for a disease with a 1-in-10,000 base rate, applied to a random patient, returns a positive result that is wrong 99 times out of 100. The arithmetic is not a trick. It is what Bayes' rule says happens when you take base rates seriously, and most human reasoning does not.
Bayesian inference is the procedure for updating a probability when new evidence arrives. It was written down by Thomas Bayes in a manuscript published posthumously in 1763, ignored for most of the next two centuries, and is now the default machinery behind spam filters, drug trials, and a sizable fraction of modern machine learning.
The move
Three quantities, one division:
P(H | E) = P(E | H) · P(H) / P(E)
- Prior
P(H)— what you believed before the evidence. - Likelihood
P(E | H)— how expected the evidence is if the hypothesis is true. - Posterior
P(H | E)— what you should believe now. - Evidence
P(E)— the normalizer; usually the hardest term to compute.
The formula is one line of algebra. The work is in picking hypotheses that are not gerrymandered, priors that are not smuggling the answer in, and likelihood models that admit how much they don't know.
Where it actually shows up
- Spam filtering. Paul Graham's 2002 essay A Plan for Spam applied a naive Bayes classifier to email and cut his spam through-rate to roughly 0.5%. Most commercial filters that followed were Bayesian variants.
- Clinical trials. The FDA has approved Bayesian adaptive designs since the mid-2000s; the I-SPY 2 breast cancer platform trial, running since 2010, uses them to decide which drugs to drop in flight.
- Search and rescue. The 1968 hunt for the lost submarine USS Scorpion used a Bayesian grid search across the Atlantic seabed. The wreck was found about 200 meters from the highest-posterior cell. The same machinery found Air France 447's flight recorders in 2011 after two years of unsuccessful conventional search.
- Sports. Nate Silver's PECOTA (baseball) and Elo-style chess and football ratings are Bayesian updates dressed in domain clothing.
The base-rate problem
Most human reasoning fails the medical-test question above. In one frequently-replicated study (Eddy 1982, repeated by Gigerenzer through the 2000s), physicians given a similar problem estimated the post-test probability at around 70-80%. The actual answer is closer to 1%. The error is so reliable that Daniel Kahneman built a chunk of Thinking, Fast and Slow (2011) on variants of it.
Bayesian reasoning is not natural. It is a posture you adopt against a strong cognitive current — see concept cognitive bias.
Priors are not cheating, but they are not free
Every inference method carries assumptions. Bayesian methods make some explicit by labeling them "prior." This is more honest, not more objective.
A prior can be:
- Informative — drawn from previous data or hard domain knowledge.
- Weakly informative — broad enough to allow surprise but rule out the absurd (a person's height is not 400 cm).
- Flat — meant to express ignorance; often does no such thing, since flatness in one parameterization is sharply peaked in another.
- Hierarchical — learned across related cases, the engine behind modern multi-level models.
When data is thin, the prior dominates. When data is abundant, the likelihood drowns the prior. Both cases require knowing which regime you are in.
What's contested
Three live arguments, none of them settled:
Frequentist vs Bayesian foundations. The 20th-century statistics establishment, led by Ronald Fisher and Jerzy Neyman, treated probability as long-run frequency and dismissed priors as unscientific. Bayesians treat probability as degree of belief. The fight was bitter through the 1970s and is now mostly a truce of mutual usefulness, but the philosophical disagreement is real.
How much computation buys how much belief. Modern Bayesian work runs on Markov-chain Monte Carlo and variational inference, both approximations. A posterior produced by 4 chains of 2,000 samples each is not the posterior; it is a sample from something hopefully near it. How much of "Bayesian" results in practice are artifacts of the sampler is an open empirical question.
Calibration in deep learning. Neural networks routinely produce confident, wrong predictions. "Bayesian neural networks" promise calibrated uncertainty; whether they actually deliver it at scale, versus cheaper ensembles or temperature scaling, is still being argued in the literature as of 2024.
Why this has to do with other realms
The deepest cross-realm bridge is to physics. Edwin Jaynes spent the second half of his career arguing that statistical mechanics — entropy, the second law, the distribution of gas molecules in a box — is best understood as Bayesian inference applied to incomplete physical information rather than as a statement about the world itself. His 1957 paper Information Theory and Statistical Mechanics and his posthumous book Probability Theory: The Logic of Science (2003) make the case. If Jaynes is right, thermodynamic entropy and the posterior of a coin flip are the same kind of object. See concept entropy and concept information theory.
An open question
If a prior chosen for mathematical convenience consistently outperforms a prior chosen from domain expertise — as happens often in modern ML — what does it mean to say the model "believes" anything at all?
Key sources
- Probability Theory: The Logic of Science by E.T. Jaynes (2003) — the book-length argument that Bayesian inference is the unique extension of logic to uncertainty.
- Bayesian Data Analysis by Gelman, Carlin, Stern, Dunson, Vehtari, Rubin (3rd ed., 2013) — the working practitioner's reference, especially on hierarchical models.
- Thinking, Fast and Slow by Daniel Kahneman (2011) — chapters on base-rate neglect and representativeness, the cognitive backdrop.
- The Theory That Would Not Die by Sharon Bertsch McGrayne (2011) — history of Bayes' rule from 1763 to U-boat hunting to clinical trials.
- Paul Graham, A Plan for Spam (2002) — the essay that pulled Bayesian classification into mainstream software.
- Eddy, D.M. (1982), "Probabilistic reasoning in clinical medicine" — the original medical base-rate finding.
Further reading
- Bernoulli's Fallacy by Aubrey Clayton (2021) — a polemical history arguing the frequentist era was a wrong turn. Worth reading even if you disagree.
- 3Blue1Brown's Bayes theorem video (YouTube, 2019) — the visualization that makes the medical-test arithmetic click in 15 minutes.
- Andrew Gelman's blog (statmodeling.stat.columbia.edu) — ongoing, opinionated practitioner notes on what Bayesian work gets wrong.
- The Signal and the Noise by Nate Silver (2012) — Bayesian updating as applied to forecasting weather, elections, and poker.
See Also
- concept probability — the underlying language.
- concept entropy — Jaynes's bridge to physics; the same math, different realm.
- concept information theory — Shannon's framework, mathematically intertwined with Bayesian updating.
- concept cognitive bias — why humans are bad at this without training.
- concept scientific method — belief revision as the engine of science.
- concept machine learning — where the priors are now mostly chosen by GPU budget.