"Stop Comparing LLM Agents Without Disclosing the Harness"
Yunbei Zhang, Janet Wang, Yingqiang Ge, Weijie Xu, Jihun Hamm and Chandan K. Reddy. arXiv
2605.23950v1, 7 May 2026, 17 pages, read from the PDF with pypdf. T1 — a position paper
with a formal statement, a controlled variance decomposition, and a proposed protocol.
It is a position paper, not a measurement, and that distinction decides what it closes: it names and formalizes what this spoke has been assembling from five separate sources, and it does not run the repeated-run experiment growth edge 12 asks for.
The Binding Constraint Thesis
Formalized rather than asserted. For a model M under harness H with benchmark score B(M, H):
- HV(M) = Var over harnesses — how much the score moves when you change the scaffold and hold the model.
- MV(H) = Var over models — how much it moves when you change the model and hold the scaffold.
The thesis is that for long-horizon tasks with comparable frontier models, HV exceeds MV. Not a slogan: two quantities, and a claim about which is bigger. That is the version of "the harness is the architecture" that can be falsified.
The numbers it assembles
| Effect | Size |
|---|---|
| Harness swap alone, Terminal-Bench 2 pass@1 | 69.7% → 77.0% |
| Scaffold-only variation, SWE-bench Verified (GPT-5) | up to 11 pp |
| Scaffold-only variation, SWE-bench Verified (Kimi K2 Thinking) | up to 15 pp |
| Single-model swing across scaffolds, HAL on SWE-bench Verified Mini | up to nearly 48 pp |
The 48-point figure is new to this corpus and is the largest harness effect it holds — against harness-bench's 23.8 pp strongest-to-weakest spread, on a different benchmark subset. And the paper reports model ranking reversal: the same model ranks above or below a competitor depending on the scaffold, which is the failure mode that makes a leaderboard actively misleading rather than merely noisy.
What it asks for
Two valid regimes, and disclosure is the precondition for both:
- Locked-harness — one specified harness applied to every model. Already instantiated by HAL across nine benchmarks, the SWE-bench Pro unified scaffold, and Epoch AI's mini-SWE-agent recommendation. The authors endorse it and name its cost: any standardized harness embeds design choices that interact with model properties, and it forecloses the harness-side gains the evidence says are large.
- Factorial — harness varied as a controlled factor, with variance components reported.
Disclosure extends past the agent loop to the infrastructure doing context construction, tool-call mediation and output verification. "Until harness specifications are disclosed, leaderboard comparisons for long-horizon agents should be treated as incomplete and potentially misleading."
The reframing worth keeping
The paper's diagram is its argument: the inference framing treats an agent as a model in a while-loop and attributes performance to the model; the closed-loop framing makes the harness a controller with its own properties — stability, context drift, control lag — and the model an open-loop policy the controller governs. On that reading, asking which model is better without naming the controller is a category error rather than an oversight. It also explains why small harness changes can outweigh a model substitution: controller properties compound over a long horizon in a way a per-token policy difference does not.
What it does not close
Growth edge 12 wants K ≥ 5 repeats per configuration with reported spread, or a reliability study
that varies the scaffold. This paper argues that such a protocol is necessary and aggregates other
people's single-run numbers to make the case. Nobody has run it. Corrected 2026-09-16 (claim
drain): the scaffold-varying half had been run before this sentence was written — beyond-pass-at-1
(arXiv 2603.29231) evaluates 396 tasks across two scaffolds with k=3 repeats, and it was already in
this spoke. The K≥5 half now has a candidate too: How Consistent Are LLM Agents? (arXiv 2605.28840)
repeats multi-step tool-calling pipelines over five seeds, though on pipelines rather than coding
harnesses and without varying the scaffold; not ingested, noted as the lead. What remains unrun is the
conjunction — K≥5 and a varied scaffold on the same tasks. So the edge half-closes: the corpus
now has the specification of the missing experiment, stated formally, and the two halves run
separately.
The thesis may be dated by construction (long-horizon-agents-survey, 2026-08-31)
long-horizon-agents-survey (Dong et al., RUC, July 2026) writes the same split as a decomposition,
Agent = π_θ ⊕ ℋ, without citing this paper — a third independent formalization of the model/harness
question inside two months, alongside model-or-harness-taxonomy. That much corroborates.
Its organizing claim does not. The survey argues the two halves co-evolve: "capabilities first implemented explicitly in the harness may later be internalized into the model policy, while stronger policies in turn enable more capable harnesses." If that is right, HV > MV is largest exactly where the harness is compensating for something the policy cannot yet do, and it shrinks wherever that compensation gets trained in. The Binding Constraint Thesis is stated here as a standing property of long-horizon tasks; under the survey's account it is a property of a moment.
The two are not in contradiction — one is a measurement on models available in 2026, the other a trajectory — and the survey measures nothing, so it cannot settle anything. What it changes is how this page's number should be read: ~48 pp is a 2026 reading, not a constant. The experiment that would decide it is the one specified below, run at two points in time. Recorded in synthesis.
Related
metr-scaffold-comparison · long-horizon-agents-survey · long-horizon-agency · model-or-harness-taxonomy · harness-bench · agent-evals · beyond-pass-at-1 · science-of-agent-reliability · claw-swe-bench · swe-bench · agentic-coding-harness · agentic-tooling · synthesis