The Harness Problem — one tool changed, sixteen models measured

Can Bölük's post behind omp's headline benchmark numbers, published 2026-02-12 on blog.can.ac and reached by chasing the "Read the full post" link on the omp README (2026-09-06). Its argument is this spoke's central thesis stated as a measurement: the model was held fixed, only the edit tool changed, and pass rates moved further than a model upgrade usually moves them.

Tier T3, and the reason matters

This is the author of the tool benchmarking his own edit format against the formats his competitors ship. Per ../QUALITY.md, a benchmark the maker ran is T3 no matter how the methodology reads. Three things soften that and none of them change the tier:

• the fixtures, harness and per-run reports are published (packages/react-edit-benchmark in the omp repository), so the run is reproducible by someone who is not the author;
• the method is stated in enough detail to attack — 180 tasks × 3 runs, fresh session per task, four tools only (read, edit, write), comparison against the original file before and after formatting;
• the results include cases that hurt the author's format (DeepSeek V3.2 loses 5 points on hashline and spends 20% more output tokens; GPT-5.2 Codex gains 4.6 and also spends 26% more).

What it is not: an independent replication, and not a measurement of anything but edit-tool success on mechanically-injected bugs.

The fixture design, and its ceiling

Tasks are generated, not collected: take a random file from the React codebase, inject a mutation framed as a bug (operator swap, boolean flip, off-by-one, a removed optional chain, a renamed identifier), then write the issue up in plain English. The inverse edit is known, so scoring is mechanical.

That design is what makes 16 models × 3 formats × 3 runs affordable — the author puts the whole spend at about $300 — and it is also the ceiling. Every bug is a single local mutation with a known inverse, so the benchmark measures whether a model can express a small edit it has already worked out. It says nothing about finding the bug, and the post is straightforward about this: "Often the model isn't flaky at understanding the task. It's flaky at expressing itself." That is the claim being tested, and the fixtures were built to test it.

The numbers

Three formats compared: apply_patch (Codex's diff blob), str_replace (Claude Code, Gemini, most others), and omp's own hashline. Pass rate per model per format, 3 runs × 180 tasks:

Headline: +15 points average over patch across 16 models; hashline beats patch in 14 of 16, and a v2 revision improves on v1 in 12 of 16 (largest gain GPT-5.1 Codex Mini, 60.0% → 77.5%). Grok Code Fast 1's 6.7% → 68.3% is the tenfold figure omp advertises.

The gradient is the finding, not the average. The weakest models gain the most and the two strongest lose ground. Read charitably that says a good edit format stops mechanical failure from hiding a model's actual coding ability, and a model already fluent in a format has nothing to recover. Read sceptically it says the benchmark's difficulty sits in a band where format dominates, and the band moves as models improve — which would make the +15 a fact about early-2026 models rather than about editing.

Why patch fails, and the mechanism proposed

The post's account of the state of the art doubles as the case for why the format is a lever: apply_patch is an OpenAI-flavoured diff whose token selection is plausibly biased at the Codex gateway, so models never trained on it fail catastrophically — Grok 4's patch failure rate measured 50.7%, GLM-4.7's 46.2%. str_replace demands perfect reproduction of existing text including whitespace, and rejects on multiple matches. Cursor's answer was to fine-tune a separate 70B model whose only job is applying edits, and even then Cursor's own post concedes full-file rewrites beat diffs under 400 lines.

Hashline's proposal: tag every line with a 2–3 character content hash on read, and let the model reference the tag rather than reproduce the content.

1:a3|function hello() {
2:f1|  return "world";
3:0e|}

The model says replace 2:f1, or insert after 3:0e. Two properties follow. It never retypes old content, which is where the token saving comes from; and if the file moved since the read, the hashes diverge and the patch is rejected before it corrupts anything — a stale-read check the other formats get only accidentally, via a string that no longer matches.

Corroboration the post itself supplies

Unusually for a vendor benchmark, it cites work that would settle the question independently, and none of it is the author's: Aider's benchmarks, where format choice alone swung GPT-4 Turbo from 26% to 59%; JetBrains' Diff-XYZ (arXiv:2510.12487), finding no single edit format dominates across models and use cases; and EDIT-Bench (arXiv:2511.04486), where only one model clears 60% pass@1 on realistic editing. The independent literature supports the weak claim — format choice is a large effect — while the strong claim, that this format is the best one, rests entirely on the author's own run. Those two arXiv papers are the obvious next sources for this spoke; neither is paged here yet.

The vendor argument, and one unverified claim

The post closes on platform access rather than measurement. It reads Anthropic's block of OpenCode from Claude Code subscriptions as signalling "don't build harnesses, use ours", and reports that Google disabled the author's Gemini account outright while the benchmark was running — not rate-limited, not warned — with a screenshot as the only evidence. That is one party's account of an enforcement action with no vendor statement beside it, and this page records it as a claim, not a fact. The argument built on it is more durable than the incident: no vendor will optimize a harness for a competitor's models, so cross-model harness tuning is work only an open harness has a reason to do. "The model is the moat. The harness is the bridge."

Re-checked 2026-10-11: still no Google statement about this account. The wider pattern is on record: between February and April 2026 Google staff on its AI developer forum acknowledged "recent bans linked to third-party tool usage" of Gemini CLI and Antigravity and announced an automated unban for affected accounts. That makes the author's account plausible. It does not confirm it.

Where it sits against what this spoke already measures

Three sources here now put numbers on harness effects and they do not measure the same object. harness-bench crosses 8 models × 6 harnesses over 106 general agent tasks; claw-swe-bench compares harnesses on coding; this post holds the harness fixed and varies one tool inside it. That is the finest-grained cut of the three and the only one performed by a party with an interest in the result. model-or-harness-taxonomy supplies the vocabulary for what it found: a failure to express an edit is a harness-side fault at the model↔tool edge, and post-training the model is the wrong repair for it.

Related

oh-my-pi · agent-edit-format · can-boluk · stencil-labs · agentic-coding-harness · harness-bench · claw-swe-bench · model-or-harness-taxonomy · agent-evals · pi-coding-agent