Capability emergence

Whether a model’s abilities arrive abruptly or gradually as training data or scale increases, and whether the abruptness is real or an artifact of how you measured. The corpus’s first source on it is phase-transitions-in-attention, which answers the question for one sub-circuit by computing it rather than observing it.

The argument it enters

Loss scaling with data and model size is understood empirically; predicting when a capability appears is not. One camp reports capabilities that emerge sharply, another argues the sharpness comes from discontinuous evaluation metrics — score a task pass/fail and a smooth improvement looks like a jump. phase-transitions-in-attention sidesteps the measurement dispute by deriving the transition analytically: in its model the discontinuity is in the posterior over attention patterns, not in a metric applied afterwards.

Two kinds of abrupt

The distinction the paper contributes, borrowed intact from statistical mechanics:

• First-order — the order parameter jumps. No precursor. Softmax attention behaves this way on the copy task, with the loss dropping sharply at a predicted boundary and no earlier learning stage.
• Second-order, then crossover — the change is continuous, preceded by diverging fluctuations, and followed by gradual sharpening. Linear attention behaves this way.

The order is set by the attention activation, not by the task, the data or the optimiser. Two architectures learning the same thing can therefore differ in whether the learning is visible before it finishes.

Why this is a monitoring question

If a transition is second-order, fluctuations grow before it and something measurable moves first, which is what progress measures and early-warning evaluations depend on. If it is first-order, the paper’s claim is blunt: the onset is “intrinsically unpredictable from observable quantities,” implying “fundamental limits to what can be monitored.” Production transformers use softmax.

That connects a piece of learning theory to AI risk-management practice — the same worry that motivates staged-capability evaluation, arriving from the mechanism side. It is one sub-circuit in a one-layer model, so the correct status is a conjecture with a derivation behind it, not a finding about frontier models.

What would settle it

A two-layer transformer with causal masking (the paper’s own next step), natural-language token statistics instead of i.i.d. uniform ones, and any measurement on a real training run showing a capability appearing with or without a precursor signal. None of that is in this corpus.

Related

phase-transitions-in-attention · induction-head · bayesian-inference · machine-learning · synthesis