HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

Wu, Zhang, Shi, Lei, Gu and fourteen others — ByteDance Seed, Singapore University of Technology and Design, Georgia Institute of Technology, M-A-P, and TokenWave.AI. arXiv:2609.01437, cs.SE and cs.CL, 2 September 2026. Arrived via Telegram; read from the PDF locally (41 pages).

Every harness measurement this spoke holds — scaffold-effects-gaia, claw-swe-bench, harness-bench, confucius-code-agent — varies harnesses that humans built. This one asks whether the model can build one, and it "shifts the unit of evaluation from task outputs to runnable infrastructure."

The design

Two stages, both starting from a deliberately weak seed — "a runnable compatibility layer, not a task-solving agent", whose tools are passive, identical for every creator, so the benchmark measures design rather than boilerplate recall.

• Creation. From that seed plus a few development cases, build a complete execution system.
• Evolution. Take your own created harness and revise it from downstream execution feedback.

Six creator LLMs: Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, Seed 2.0 Pro. Development happens inside Claude Code 2.1.177 (GPT-5.5 uses Codex 0.144.3). Four domains, five benchmarks, 2,207 unique downstream instances: SWE-Pro and Terminal-Bench 2.1 (code), MLE-bench (ML experimentation), EQ-Bench3 (writing), BrowseComp (search and research). Three harnesses per creator–benchmark pair, reported as avg@3, with hidden tasks withheld from the development loop.

Two evaluation modes, and the difference between them is a finding: Self-Eval runs each harness on the model that wrote it; Unified-Eval runs every harness on Gemini 3.1 Pro.

Creation: the gap is domain-shaped, not uniform

Overall scores, Self-Eval:

The aggregate hides the interesting part. Against the human reference per benchmark:

So a model-built harness beats the human reference by nearly nine points on ML experimentation and by a point on writing, and loses by forty on search. The paper's reading is that the gap is largest where long-horizon information seeking is required. The corollary nobody states as sharply: the domains where models win are the ones where the harness has least to do.

And the harness, not the model, is where the failures live. "77.8% of failed Data tasks are attributed to harness defects, showing that the bottleneck is not only executor capability."

Evolution: +4.44 points, at best, for one model

"All five self-runtime creators improve on the visible feedback pair, but the gains shrink on held-out tasks. Opus 4.8 has the largest held-out improvement at +4.44 points." Under the fixed Gemini executor, only Opus improves on held-out tasks; the other three lineages regress.

The paper's own summary is the one to quote: "Performance often rises and falls across successive revisions, and gains observed during development become smaller and less consistent on unseen tasks."

This is the measurement autoharness said did not exist. That page has carried, since ingest, that whether a self-maintained skill layer stays net-positive over time is "the open self-improvement question, still unmeasured here." It is measured now, for harnesses rather than skill libraries, and the answer is a small unstable positive that mostly does not transfer. Set that beside autoharness's own unverified 42% → 78% headline: the best result in a controlled 2,207-instance study is +4.44 points on held-out tasks, for one of six models.

What it does to the spoke's harness thread

harness-bench found that harness sensitivity falls as the model gets better — stronger backends score higher with lower cross-harness variance. HarnessDev looks like a tension and is not quite one: "Fixing the executor changes the ranking substantially. Qwen, Seed, and DeepSeek improve in several Data and Search settings under Gemini, whereas Opus and GPT-5.5 are often stronger with their own executors." Harness-Bench measured which harness is better; this measures compatibility between a harness and the model running it, and finds that Self-Eval scores conflate three things — harness design, executor capability, and the fit between them. That is a fourth variable the spoke's earlier factorials did not separate.

It also supplies the motivating number for the whole thread in one line, cited from the Terminal-Bench work: with identical weights, GPT-5 solves 35.2% of Terminal-Bench 2.1 inside Terminus 2 and 49.6% inside Codex CLI.

Two things to read carefully

The human baseline is not a controlled arm, and the paper says so: "Human-reference results are verified public system results rather than paired controls under one executor." The references are each a different published system on a different model — a public coding-agent setup on Claude Fable 5, an OpenAI agent setup on GPT-5.6 Sol, MLEvolve on Gemini 3.1, Kimi Writer on Opus 4.8, the OpenAI browsing stack on GPT-5.6 Sol. So "86.2 versus 67.8" compares a model-built harness against a best-available published result, not against a human harness run under identical conditions. The direction of the finding survives that; the size of the gap is not a clean number.

The affiliation cuts the unusual way. ByteDance Seed leads the author list and Seed 2.0 Pro is one of the six creators — placing last, at 22.8, less than half the leader. A vendor benchmark where the vendor's own model finishes bottom is the opposite of the pattern the hub's "nobody re-runs anybody's numbers" item keeps finding. It does not make the benchmark independent, and it does remove the obvious reason to distrust the ranking.

Not read: Appendix B (configurations, resource limits, reference sources), the edit statistics and trajectory analysis, and Table 4's Unified-Eval cells.

Related

agentic-coding-harness · harness-bench · scaffold-effects-gaia · claw-swe-bench · confucius-code-agent · model-or-harness-taxonomy · self-improving-agents · autoharness · agent-evals · longhorizon-harness · synthesis