Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows

Yao, Tan, Liu et al., arXiv:2605.27922 (2026-05-27). The full crossed factorial claw-swe-bench declined to run, on general agent work rather than coding — and it closes the question that paper left open at the top of the model range. Arrived via the Quality Cycle’s research pass, against this spoke’s harness spread at the frontier edge.

The design

106 sandboxed offline tasks, built from practical agent-use patterns and manually reviewed for “realism, solvability, oracle-checkability, and integrity.” 5,194 execution trajectories across a grid of 8 model backends × 6 harnesses, plus Codex as a model-bound coding agent on GPT-5.4.

• Models: claude-opus-4.6, claude-sonnet-4.6, gemini-3.1-pro-preview, gpt-5.4, glm-5.1, qwen3.6-plus, kimi-k2.5, deepseek-v4-flash.
• Harnesses: openclaw, ZeroClaw, Hermes, Moltis, NullClaw, NanoBot.
• Held fixed: task environments, budgets and evaluation protocol — while “preserving each harness’s native execution behavior,” which is the hard part of a comparison like this.

Each run records final artifacts, execution traces, usage statistics and validator output, so the analysis goes past pass/fail into process quality, efficiency and failure mode.

What it found

The harness gap is large, and it is not where this wiki’s attention has been. Aggregate scores run from NanoBot 76.2% (81.6% completion) down to openclaw 52.4% (60.0% completion) — a 23.8-point spread between the strongest and weakest harness. Process scores span 79.5%–93.8%, and all six harnesses held 100% security compliance.

And the answer to the open question: “Stronger model backends tend to achieve higher mean scores while exhibiting lower cross-harness variance.” Harness sensitivity falls as the model gets better.

Why that settles something here

claw-swe-bench found harness spread roughly doubling as the model weakens (12.5 pp on GLM 5.1 → 27.4 pp on Qwen 3.6-flash) and could not test the top of the range — it ran two grids against two mid-tier models. This wiki recorded that untested cell as the live question: does the spread keep shrinking toward zero at the frontier, making harness engineering a crutch, or does it plateau, making it a permanent layer?

Harness-Bench puts frontier models in the grid and the trend continues: the better the model, the less the harness matters. Two things follow, and they pull in opposite directions.

• Scaffolding substitutes for capability (scaffold-effects-gaia‘s finding, now confirmed at three points on the capability axis) — which means much of what this spoke documents is priced in model progress.
• It has not reached zero. A 23.8-point aggregate spread across harnesses is not a rounding error, and the authors’ conclusion is that “agent capability should be reported at the model-harness configuration level rather than attributed to the base model alone.” Reporting a model’s agentic score without naming the harness remains meaningless.

The uncomfortable detail: openclaw — the most-starred harness in this corpus and the one most of its sources orbit — scores last of six. The star count measures adoption and this measures execution, and the spoke has said the former is not evidence of the latter (qm‘s reminder). Here is the case where they visibly disagree.

Its own recorded limits

Four, from the authors: sandboxed offline workflows, so no live-service or real-time feedback; complete harness configurations rather than isolated mechanisms, so it says which harness wins and not which mechanism won it; rubric-based and LLM-assisted process scoring; and results that are “diagnostic measurements under fixed protocol, not deployment guarantees.”

The second of those is the one that matters for this wiki. The corpus documents mechanisms — memory, middleware, guardrails, orchestration — and this benchmark scores bundles. It cannot tell you which of NanoBot’s choices bought the 23.8 points.

Related

claw-swe-bench · scaffold-effects-gaia · agentic-coding-harness · openclaw · hermes-agent · agent-evals · model-or-harness-taxonomy · qm · synthesis