Towards a Science of AI Agent Reliability (ICML 2026)

Rabanser, Kapoor, Kirgis, Liu, Utpala and Narayanan (Princeton), submitted 2026-02-18, accepted at ICML 2026. 14 proprietary frontier models, two benchmarks, K=5 independent runs per task at temperature zero, twelve metrics across four dimensions. The result the spoke had been asking for since beyond-pass-at-1: repeated-run reliability measured on the models people actually deploy.

Despite steady accuracy improvements over 18 months of model releases, reliability only shows modest overall improvement.

What it measures, and why the metrics are not accuracy in disguise

Four dimensions, twelve computable metrics: consistency (outcome, trajectory-distributional, trajectory-sequential, resource), robustness (fault, environment, prompt), predictability (calibration, AUROC discrimination, Brier), safety (compliance, harm severity).

The design decision that makes them usable is disentanglement from capability. Outcome consistency normalizes variance by p(1−p) — the maximum variance available at that success rate — and the robustness metrics are ratios between perturbed and nominal conditions rather than absolute scores. So a weak model is not penalised twice, and two agents at the same accuracy can be compared on how they fail. Every other reliability number this spoke holds is entangled the way the authors are avoiding.

Setup: GAIA (165 validation tasks, three difficulty levels) and τ-bench, restricted to the 26 tasks left after Cuadron et al.'s grading errors are removed. Models span April 2024 to December 2025 across OpenAI, Google and Anthropic — GPT-4o mini through GPT-5.2, Gemini 2 Flash through Gemini 3 Pro, Claude 3.5 Haiku through Claude 4.5 Opus — run through Princeton's HAL harness with cost tracking. No open-weight model is included, which is the exact complement to beyond-pass-at-1's open-weight-only design.

Findings

Outcome consistency stays low everywhere, frontier models included. Agents that can solve a task often fail to do so across five runs, which is the same thing as the gap between pass@k and pass^k that tau-bench introduced — measured here across a model generation rather than at one point.

"What but not when." Distributional trajectory consistency is substantially higher than sequential: agents pick similar action types run to run and vary the order. So the instability is in planning and execution rather than in action selection, which is a more specific diagnosis than this corpus has had for agent nondeterminism.

Consistency runs the wrong way with model size. Calibration, robustness and safety improve with capability inside a family, but smaller models are often equally or more consistent than the larger ones in the same family — the authors' reading is that a bigger model has more ways to solve a task, so run-to-run variance grows. Reasoning models are generally, though not uniformly, more reliable, and their reliability improves more slowly than their accuracy.

Resource use is unpredictable, and worse on hard tasks. Token and compute consumption varies widely across identical runs, especially on GAIA, and degrades further with difficulty — with the newest Claude models showing a "try harder" pattern of elevated action counts on hard tasks. That is the first measurement in this spoke of the cost variance operators actually pay (effort-level, agent-storage-footprint).

Where reliability improved, and where it did not. Gains are concentrated in the structured environment: τ-bench shows moderate improvement over the 18 months, GAIA barely any. Discrimination — telling one's own correct answers from incorrect ones — has generally worsened on GAIA even as calibration improved.

Safety on τ-bench: financial-accuracy violations are the most common failure; high-severity violations are rare across frontier models, and state-of-the-art models across all three vendors have the lowest violation rates.

What it does to the spoke's arguments

beyond-pass-at-1 found long-horizon reliability decaying on open-weight models and its own limitations section guessed that "GPT-4o, Claude and Gemini are likely more reliable than the models studied here." This paper runs the proprietary set and reports modest improvement over 18 months of releases, with the open-ended benchmark flat. The guess is not supported: the reliability plateau is not an artifact of studying open models.

It also sharpens the agentic-coding-harness thesis rather than confirming it. The spoke's claim is that structure around the model does the work; this paper measures models under one fixed scaffold per benchmark and finds the failure is in stability of planning and execution — which is where a harness would intervene. Nobody has crossed the two: scaffold-effects-gaia varies the scaffold with three attempts, this varies the model with five runs, and no study varies both with repeats.

For agent-evals the recommendation is blunt and aimed at how this corpus's own benchmark pages were built: single-run accuracy on a fixed environment "reveals nothing about whether an agent would succeed on the same task tomorrow." The authors ask for multi-run protocols plus generative, parameterized benchmarks that rename fields, reorder responses and inject faults.

The retrospective table is the part an operator can use: Replit's database deletion maps to harm severity plus prompt robustness (would the "do not delete the database" constraint hold under rephrasing?), OpenAI Operator's unconfirmed purchase to compliance plus trajectory consistency, the NYC chatbot's confident wrong legal guidance to calibration plus outcome consistency. The claim that each "could have been identified prior to deployment" is the authors' own reading of failures they did not observe, and it is the least checkable thing in the paper.

Limits the authors state

Two benchmarks only; one scaffolding approach per benchmark; LLM-based safety judging, which has its own reliability problem; metric choice is one decomposition among several; safety is reported separately rather than aggregated. They also answer the obvious objection — that a sufficiently capable model would be reliable anyway — with the argument that this holds only at perfect accuracy, and that at equal accuracy one agent may fail on a fixed identifiable subset while another fails unpredictably on a different subset each run. Only the first can be debugged and guard-railed.

Related

beyond-pass-at-1 · tau-bench · scaffold-effects-gaia · agent-evals · harness-bench · agentic-coding-harness · effort-level · agent-storage-footprint · swe-bench · sayash-kapoor · arvind-narayanan · princeton-university · synthesis