Refusal in Language Models Is Mediated by a Single Direction

Arditi, Obeso, Syed, Paleka, Panickssery, Gurnee and Nanda, arXiv 2406.11717 — submitted 2024-06-17, three versions through 2024-10-30. The paper abliteration rests on, and the one every abliteration tool in the field cites, including heretic.

Ingested here from the abstract and from heretic's account of the method; the full text has not been read, so the claims below are the ones the authors state, not ones this wiki has checked against their evidence.

The finding

Refusal is one-dimensional. Across 13 open-source chat models up to 72B, the authors find a single direction in the residual stream such that erasing it stops the model refusing harmful instructions, and adding it makes the model refuse harmless ones. The second half is the part that makes it a mechanism rather than a correlation: the direction is not merely present when the model refuses, it is sufficient to cause refusal where none was warranted.

From that they build a white-box jailbreak that "surgically disables refusal with minimal effect on other capabilities," and a mechanistic account of how adversarial suffixes work — by suppressing the propagation of that same direction.

Why the conclusion is uncomfortable

The authors' own summary: the results "underscore the brittleness of current safety fine-tuning methods." Safety training that costs a lab a great deal of compute deposits its result in a subspace you can find with a difference of means over two prompt sets, and remove with a matrix orthogonalization. Nothing about the training makes the result hard to locate.

The paper is dated 2024. heretic, in 2026, is that finding turned into a pip install with a parameter search on top — and the numbers say the surgery got cleaner over those two years, not that it got harder. This spoke's open-weight-models page treats a permissive licence as permission to rewrite the weights; this is the paper explaining why the rewrite is cheap.

The seam it names

"Our work showcases how an understanding of model internals can be leveraged to develop practical methods for controlling model behavior." Read forwards that is a steering result; read backwards it is a jailbreak. The paper says both, in the same sentence, and does not pretend the two uses can be separated — which is the same seam abliteration and heretic land on from the tooling side.

Related

abliteration · heretic · gemma-4-31b-it-scotoma · open-weight-models