Measuring Time Horizon using Claude Code and Codex (METR, 2026-02-13)
A METR research note by Nikola Jurkovic asking whether a vendor's own coding harness lifts a model's task-length horizon above what METR's generic scaffolds measure. It is the closest thing the corpus holds to the factorial model × harness experiment harness-disclosure specified: two models, each run under its vendor's product harness and under a plain METR scaffold, on the time-horizon task suite.
The cells
| model | product harness | generic scaffold | product beats generic in bootstrap samples |
|---|---|---|---|
| Opus 4.5 | [[claude-code | Claude Code]] | ReAct |
| GPT-5 | Codex | Triframe | 14.5% — the product harness loses |
Neither product harness outperforms METR's default. The result is sharper than "no difference": Codex under-performs a generic scaffold on the same model, with the 14.5% saying the generic one is better in most resamples.
Why, in METR's reading
Claude Code and Codex "are prompted much more elaborately than either Triframe or ReAct, including to use to-do lists for long tasks," and are tuned for their model families. What they are tuned for is an interactive session with a human in the loop. Remove the human and the tuning stops paying: GPT-5 under Codex tried to talk to users who were not there; Opus 4.5 under Claude Code anchored on its first plan and would not pivot, and at times spent resources inefficiently. The note's conclusion is that the product harnesses were built for collaboration, not for the unattended runs METR scores.
What it does and does not settle
- For harness-disclosure's thesis — that variance over harnesses exceeds variance over models — this is a counter-instance at the top of the capability range: on these two models the harness moved the horizon by roughly nothing. The ~48 pp swings that paper aggregates came from weaker models and from harnesses that differ more than "product vs ReAct".
- For spoke growth edge 12 (reliability with the harness as the variable) it is a half close. The factorial design was run — model × harness, two levels each — and the uncertainty is reported, but as bootstrap resampling over tasks, not as ≥5 seeded repeats per cell with a variance decomposition. Whether a K≥5 rerun would find the same coin flip is exactly what the edge still asks.
- For long-horizon-agents-survey's claim that the decisive experiment had never been run: it has now been run once, at one point in time. The longitudinal half — the same cells measured again later to see the harness gap close as capability is trained in — remains unrun, and long-horizon-agency's sentence about that stands.
- Two models, four cells. Nothing here generalises to weaker models, where the co-evolution account predicts the harness gap should be largest.
Found on 2026-09-14 while draining the absence-claim queue, not by a hunt; ingested against edge 12.
Related
metr · harness-disclosure · long-horizon-agency · long-horizon-agents-survey · claude-code · agentic-coding-harness · beyond-pass-at-1 · synthesis