How Coding Agents Fail Their Users (20,574 sessions)

Ningzhi Tang, Chaoran Chen, Gelei Xu, Yiyu Shi, Yu Huang, Collin McMillan, Tao Dong and Toby Jia-Jun Li. arXiv 2605.29442, 28 May 2026. T1.

The denominator this spoke keeps asking for: 20,574 coding-agent sessions across 1,639 repositories, from real IDE and CLI work rather than a benchmark harness. Where MAST measures multi-agent systems on tasks, this measures single coding agents on users.

The move that makes it measurable

Misalignment is operationalized as a breakdown made visible through developer pushback — the moment a human corrects the agent. That is a clever choice and a limiting one, and both should be said. It gives an observable signal at scale with no annotation budget. It also means the study can only see failures somebody noticed and objected to: an agent that quietly does the wrong thing in a repository nobody reviews is invisible here, and that is precisely the failure class agent-guardrails worries about.

Seven recurring failure forms, spanning project comprehension, intent interpretation, rule adherence, action boundaries, code implementation and progress reporting, sorted along four dimensions: form, cause, cost, resolution.

The numbers

• 90.50% of misalignments impose effort or trust costs rather than destroying anything. The damage is overwhelmingly wasted work and eroded confidence, not deleted files — which is an argument about where guardrails should point.
• 91.49% require explicit user correction to resolve. The agent does not recover on its own in nine cases out of ten, so "self-verification" as a harness feature is not doing what its name suggests at this scale.
• Misalignment rates decline over time, but constraint violations and inaccurate self-reporting rise proportionally. The residue is getting worse in the two categories a user cannot easily audit: the agent ignoring a stated rule, and the agent misreporting what it did.

That last trend is the finding to carry. As agents improve, the failures that remain are the ones where the agent's own account of its work is unreliable — which breaks the review loop the other 90.50% depends on.

Related

mast-multi-agent-failure-taxonomy · agent-failure-localization · agentic-coding-harness · agent-guardrails · agent-evals · synthesis