Bayesian inference

Treating the parameters of a model as unknowns with a probability distribution, and treating learning as the act of updating that distribution when data arrives. The corpus’s first source on it is information-theory-inference-learning-algorithms, where it is not one technique among many but the frame the whole second half of the book is written in.

Learning is inference

The connection that matters for this spoke is mechanical, not philosophical. Take the objective a neural network minimizes — an error term plus a regularizer, M(w) = G(w) + E_W(w) — and read each piece as a log probability information-theory-inference-learning-algorithms:

the error function is minus the log likelihood: P(D|w) = exp(−G(w));
the regularizer is a log prior; a sum of squared weights corresponds to a Gaussian prior over the weights with variance 1/α;
so minimizing M(w) finds the most probable parameter vector given the data, w_MP.

MacKay’s reason the mapping works at all: error functions are additive, probabilities are multiplicative, and the logarithm is what carries one into the other. Nobody has to choose to be Bayesian for this to hold — a trained network already is one, with a prior it may not know it has.

Point estimate versus ensemble

Where the two views part company:

“The product of traditional learning is a point in w-space, the estimator w*, which maximizes the posterior probability density. In contrast, in the Bayesian view, the product of learning is an ensemble of plausible parameter values.”

That ensemble is what makes predictions honest about their own uncertainty, and it is also what is usually impossible to compute. The rest of the machinery — Laplace’s approximation, variational bounds, and monte-carlo-methods — exists to approximate an integral over it.

Occam’s razor comes for free

Model comparison in this frame does not need a complexity penalty bolted on. The evidence P(D|H) is a normalized distribution over data sets, so a model flexible enough to predict many possible data sets must predict each one less strongly. A simpler model that concentrated its predictions where the data actually landed wins the ratio outright.

“Coherent inference (as embodied by Bayesian probability) automatically embodies Occam’s razor, quantitatively.”

This is the corpus’s first account of why anything about model selection works, as opposed to which knob to turn. classical-ml-algorithms still has no comparable account for the non-Bayesian machinery.

Where the two views were made to meet (2026-08-07)

phase-transitions-in-attention does the thing the book describes and the toolchain skips: it marginalises the weights of an attention layer and works with the resulting posterior over attention patterns, reduced by symmetry to two scalar order parameters. The reduction is what makes it possible — nobody integrated over billions of parameters; a permutation symmetry did the work.

Two results matter for this page. The posterior’s MAP is where Adam-trained networks land, and the paper derives why: minimising the negative log-posterior at large χ corresponds to gradient flow with weight decay. Posterior sampling (SGLD) and ordinary training agree closely in its figures. So in one tractable setting the point estimate everyone ships is not a different activity from Bayesian learning — it is the mode of the posterior, reached by descent.

The effective landscape also carries something a loss surface over weights does not: it “accounts for the number of microscopic weight configurations that produce the same attention pattern,” which is the entropy term Occam’s razor is made of, applied to circuits rather than models.

Scope: one layer, one head, a synthetic copy task, and asymptotic approximations. It does not show the correspondence survives at scale — it shows there is something to check.

What the corpus does not have

Nothing here on the practical cost of Bayesian methods at scale, which is the standing objection to them: posterior sampling over a modern network’s parameter count is not something anyone does, and the fine-tuning sources reach for point estimates without discussing the trade. The book predates that scale entirely, so it cannot arbitrate, and the paper above buys tractability with a symmetry that a real network does not have.

information-theory-inference-learning-algorithms · phase-transitions-in-attention · capability-emergence · monte-carlo-methods · information-theory · machine-learning · classical-ml-algorithms · synthesis