Claw-SWE-Bench: Evaluating OpenClaw-style Agent Harnesses on Coding Tasks

Zheng et al., arXiv:2606.12344v1 (2026-06-10). The coding-domain answer to the question scaffold-effects-gaia answered for general-assistant work: how much does the harness matter next to the model, measured on SWE-bench-class tasks rather than on GAIA.

The design

350 GitHub issue-resolution instances across 8 languages and 43 repositories. Held constant for every run: the prompt template, the Docker environment and repository state, a 3,600-second per-instance timeout, and the patch-extraction and SWE-bench evaluation path. So a difference in the result is a difference in the harness or the model, not in the task or the budget.

Not a full factorial — the authors run two grids and say why (cost, interpretability):

• Model sweep — openclaw fixed, 9 models varied.
• Claw sweep — 5 harnesses × 2 representative models (GLM 5.1, Qwen 3.6-flash).

The five harnesses span implementation languages as well as designs: openclaw (Node.js, per-instance isolation), hermes-agent (stateless Python), zeroclaw (a Rust binary), nanobot (Python with session logging), and a generic wrapper as the floor.

The numbers

Model sweep, OpenClaw fixed (Pass@1 of 350):

Model spread: 29.4 pp.

Claw sweep (Pass@1 of 350):

Harness spread: 12.5 pp on GLM 5.1, 27.4 pp on Qwen 3.6-flash.

What it settles

The harness is a first-order factor, and the two axes are the same order of magnitude. Swapping the harness under a fixed model moves Pass@1 by up to 27.4 pp; swapping the model under a fixed harness moves it 29.4 pp. The authors’ warning is the practical one: that is “large enough to reorder leaderboard conclusions if the harness is not specified” — so a coding-agent score reported without its harness is close to meaningless, which is how most of them are reported.

The harness matters more when the model is weaker. 12.5 pp of spread on GLM 5.1 against 27.4 pp on Qwen 3.6-flash. The two rows that move most are the two weakest harnesses: generic loses 24.5 pp going from GLM to Qwen while openclaw loses only 7.4. A good harness is partly compensating for the model — scaffolding substitutes for capability, and the substitution is worth more the less capability there is.

This matches what scaffold-effects-gaia found on general-assistant tasks, which makes the result look domain-independent rather than a quirk of either benchmark. Two benchmarks, different domains, same conclusion: the harness layer is where the engineering is.

Caveats

• The two grids are not a crossed factorial. Only 2 of the 9 models were run against all 5 harnesses, so a model × harness interaction term is estimated from two columns, not fitted. The direction of the interaction is clear; its size for the frontier models is not measured.
• Both sweep models are mid-tier. Neither GPT 5.5 nor Claude Opus 4.7 appears in the claw sweep, so whether harness spread keeps shrinking at the top of the model range is exactly the untested cell.
• One ordering artifact in the published model table (DeepSeek-V4 Flash listed below MiniMax 2.7 despite scoring higher); the per-row figures are internally consistent — every Pass@1 matches its resolved count over 350.
• Harnesses are OpenClaw-style; SWE-agent and OpenHands are not among them.

Related

• agentic-coding-harness — the layer this measures
• scaffold-effects-gaia — the same question on GAIA; this is its coding-domain counterpart
• openclaw · hermes-agent — two of the five harnesses evaluated
• synthesis

The cell it could not test, now tested (2026-08-08)

harness-bench runs the full crossed factorial this paper declined (8 models × 6 harnesses, 106 tasks, 5,194 trajectories) with frontier models in the grid, and the trend found here continues at the top: “Stronger model backends tend to achieve higher mean scores while exhibiting lower cross-harness variance.” So harness spread keeps narrowing as capability rises — without reaching zero, since its strongest-to-weakest harness gap is still 23.8 points.