Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents

Aaditya Khanal, Yangyang Tao and Junxiu Zhou, School of Computing and Analytics, Northern Kentucky University, arXiv 2603.29231, 31 March 2026. T1. Read from the PDF, extracted locally.

The paper this spoke's benchmark thread has been missing. 396 tasks × 4 duration buckets × 3 domains (33 per cell), 10 models, two scaffolds, k=3 repeats, 23,392 episodes — the first source in the corpus that runs agents repeatedly and reports the spread rather than a single score. Found by the research pass hunting the confucius-code-agent successor edge: an ablation with repeated runs and reported variance, covering persistent memory. It supplies all three.

Its framing claim is one sentence: benchmarks measure capability — does the agent succeed once — while deployments need reliability, whether it succeeds consistently. The paper's argument is that these two diverge as tasks get longer, and that pass@1 on short atomic tasks is structurally blind to the divergence.

The blindspot, stated as a table

Its Table 1 is the cleanest indictment of the benchmark set this spoke already pages. Columns are duration as an independent variable, inter-run variance measured, partial credit:

swe-bench and webarena "evaluate a single attempt per issue and do not study variance." That is the flaw underneath agtx's n=1 benchmark table, named here as a property of the whole evaluation regime rather than of one author's pilot.

Four metrics

• RDC — Reliability Decay Curve. How pass^k degrades across duration buckets.
• VAF — Variance Amplification Factor. How much duration amplifies stochastic failure.
• GDS — Graceful Degradation Score. Partial credit for agents that get part-way through a long task, which pass/fail throws away.
• MOP — Meltdown Onset Point. Behavioural collapse detected by sliding-window entropy over the tool-call sequence — the agent's actions stop looking like a strategy. Note this is a measure over the trace, not the output, and it is the one metric here that a harness could compute live.

What it found

Decay is domain-stratified, not general. Software engineering GDS falls 0.90 → 0.44 across the duration range; document processing is nearly flat (0.74 → 0.71); web research runs 0.71 → 0.40. "Long-horizon agents degrade" is too coarse a claim — it depends on what the long horizon is made of.

High variance is a capability signature. VAF splits cleanly by tier: frontier ≥ 2.37, mid-tier ≤ 1.26. The counterintuitive reading is the paper's own — the more capable models are the more variable ones, because they attempt ambitious multi-step strategies that sometimes collapse. The same mechanism produces finding four: the strongest models have the highest meltdown rates, up to 19%, which the authors call the "MOP paradox."

Capability and reliability rankings diverge, with multi-rank inversions between medium and very-long horizons. A leaderboard position earned at short horizons does not survive to long ones.

Per-bucket pass@1 with 95% confidence intervals (ReAct scaffold, k=3) — the corpus's first agent table with error bars on it:

Memory scaffolds do not help, and the abstract overstates it

This is the finding the edge asked for, and the source contradicts itself about its size.

The abstract says: "memory scaffolds universally hurt long-horizon GDS (negative or neutral for all 10 models), providing strong evidence against naive episodic memory as a reliability intervention."

The body says: the memory scaffold reduces long and very-long GDS relative to ReAct for 5 of 9 paid models, worst on the strongest ones — Llama 3.3 70B at Δ = −0.13, Qwen3 32B at Δ = −0.09 — and Mistral Nemo shows a small positive effect, Δ = +0.05.

Five of nine is not universal, and one model improving is not "negative or neutral for all 10." Both claims are recorded; the body's number is the one to cite. The direction survives the correction and the mechanism the authors propose is worth keeping: the scratchpad's extra tool calls and context injection "tax capable models' step budgets more than it benefits them" — a cost charged against a finite budget, which is context-rot's argument arriving from the reliability side.

For this spoke it is the fourth negative result on added structure, and the first about memory specifically. cc-thinking-skills, context-files-ablation-paper and reasoning-scaffolds found process effects without correctness effects; this one finds a correctness effect pointing the wrong way, against the mechanism agent-memory and self-improving-agents both rest on. Naive episodic memory is not the only kind, and nothing here tests the curated or retrieval-gated variants — but the burden has moved.

Cost, which matters for whether anyone repeats this

The full 23,392-episode study cost ≈$40.83 for the short and medium buckets, with all four estimated at $80–120 — roughly $0.004–0.005 per episode, all through OpenRouter. Set that against jetbrains-ponytail-tested's $246.09 for 251 trials on one skill. Reliability evaluation at this scale is not gated on budget, which makes its absence from swe-bench and the rest a choice about what to measure rather than a resource constraint.

Limitations, stated by the authors

• Open-source only. All 10 models are open-weight, accessed via OpenRouter; GPT-4o, Claude and Gemini are not evaluated and are "likely more reliable than the models studied here." So "frontier" throughout means the frontier of this set.
• Duration is a proxy. Task difficulty is estimated human completion time, and the paper shows the proxy breaking: document-processing tasks classed "long" by human time need only 4–8 agent tool calls.
• Infrastructure reliability is a validity threat, and this one is a finding in its own right. Three free-tier models exhausted daily API quotas after the short tasks and before the long-horizon ones, inflating apparent pass@1 through systematic selection bias; two more returned HTTP 404 all run for provider capacity reasons despite being listed. All five were replaced with paid multi-provider routing. The authors' recommendation deserves adopting anywhere in this corpus: treat completion rate — the fraction of planned episodes that ran to termination without infrastructure error — as a first-class validity metric.
• Three domains (SE, web research, document processing); no embodied action, tool fabrication or multi-agent coordination.

The complement arrived (2026-08-23)

science-of-agent-reliability (Princeton, ICML 2026) runs the design this study could not: 14 proprietary frontier models, no open-weight ones, K=5 at temperature zero, on GAIA and the clean τ-bench subset. It answers the limitation stated here directly. The guess above — that GPT-4o, Claude and Gemini are "likely more reliable than the models studied here" — does not hold: reliability improves only modestly across 18 months of proprietary releases, and hardly at all on GAIA.

One finding of this page gets independent support from a different metric. High variance as a capability signature (frontier VAF ≥2.37 against mid-tier ≤1.26) reappears there as consistency running backwards against model size within a vendor family, with the same proposed cause: a more capable model has more ways to attempt the task. Two designs, two metrics, one direction.

Related

science-of-agent-reliability · agent-evals · swe-bench · tau-bench · webarena · agtx · agent-memory · self-improving-agents · reasoning-scaffolds · cc-thinking-skills · context-files-ablation-paper · confucius-code-agent · context-rot · jetbrains-ponytail-tested · agent-specific-task-difficulty · harness-bench