Agent failure localization

Deciding which component of an agent system caused a failure, so you know which repair to attempt. Distinct from failure detection (something went wrong) and failure classification (what kind of wrong): localization asks whose fault, because the answer selects between four unrelated interventions — model post-training, harness engineering, environment redesign, or benchmark repair model-or-harness-taxonomy.

Why outcome labels are not enough

An agent ignores an instruction given earlier in a long session. Either the harness's context compaction removed it, or it was still present and the model didn't follow it. The trace shows the same behaviour; the fix is in different codebases model-or-harness-taxonomy. This is the repair-assignment problem, and it is the reason a score on a benchmark — the thing agent-evals produces — does not tell a team what to change next.

The mechanism: interaction as the unit

Scale AI's taxonomy makes the interaction between two components the thing being labelled, not the component itself. Each failure gets an edge (which two components were interacting) plus a fault side (which of the two is responsible), written COMP1—COMP2 · fault:SIDE. A tool wrapper that swallows an error and a model that ignores a returned error sit on the same edge with opposite fault sides, and want opposite fixes.

Two design choices carry most of the weight:

• A fixed root-cause rule. Trajectories cascade, so without a rule two annotators label the same trace differently. The rule: trace backward from the observed failure to the earliest failure from which execution does not recover; everything after is a symptom.
• Roles instead of new components for multi-agent cases — a peer or subagent is another model, so it sits on a MODEL—MODEL edge with a role, and the vocabulary doesn't grow to cover orchestration.

Where the fault lands, and the bias in that

Of 41 modes, 36 are model-side — but by construction: fault is model-side whenever a stronger model could have avoided or recovered from the failure. The five non-model modes are the load-bearing ones, since they mark failures that no model upgrade fixes model-or-harness-taxonomy. The same bias reappears in practice: LLM judges applying the taxonomy over-attribute to the model when a task fails.

This is the mirror image of the spoke's own standing claim. agentic-coding-harness argues value has moved from the model to the scaffolding around it; a taxonomy built to locate blame nevertheless defaults blame to the model. Both can be true — structure decides how work gets done, while the capability ceiling stays the model's. The spoke has no measurement that settles it either way.

Relation to the spoke's other failure lists

The spoke already holds failure inventories, and they are doing a different job:

• agent-anti-patterns lists nine build-time mistakes (too many tools, no memory architecture, no observability). Prescriptive, before the fact.
• context-rot and cca-f-lost-in-the-middle describe one failure mechanism in depth.
• Localization is post-hoc and diagnostic: it takes a trace that already failed and assigns it.

The practical link is observability. Assigning fault to an edge requires a record of what crossed that edge, which is the case agent-evals already makes for tracing — the score tells you that you regressed, the trace plus a localization label tells you where to look.

One component now has its own localization method (2026-08-15)

agent-skills-can-be-harmful does for the loaded skill what the taxonomy above does for components in general, and it solves the attribution problem differently: instead of labelling an edge from a single trace, it runs the same task twice with only the skill setup varied. A failure is charged to the skill only when a no-skill or matched-skill reference run passes, or passes cheaper. The reference run is a pseudo-oracle, not a solution — it establishes that the task was solvable under this verifier, which is the whole of what attribution needs.

The method generalizes further than the result does. Paired execution buys certainty that trace labelling cannot, and it costs a second full run per audited case plus a benchmark with a deterministic verifier — so it localizes to a configuration knob (skill on / skill off / other skill) rather than to an arbitrary component. Anything you can toggle and re-run is in reach; the model's own reasoning is not.

The frequencies question below stays open, and the paper does not close it: its 307 cases are confirmed harms, with no denominator reported.

Open

Nobody has published frequencies. Closed 2026-08-30, and it was already false when it was written on 2026-08-06. Two studies publish exactly this distribution, and the older of them predates the claim by seventeen months.

• MAST (Cemri, Pan, Yang et al., arXiv 2503.13657, NeurIPS 2025) — 14 failure modes in 3 categories, 150 traces hand-annotated at κ = 0.88 and then 1,600+ traces via a validated LLM judge across 7 frameworks, cut by model and by task. System design issues 44.2%, inter-agent misalignment 32.3%, task verification the rest, with the individual modes published: step repetition 15.7%, unawareness of termination 12.4%, disobey task specification 11.8%.
• coding-agent-misalignment (Tang et al., arXiv 2605.29442, May 2026) — the denominator, at 20,574 real sessions across 1,639 repositories: 90.50% of misalignments cost effort and trust rather than destroying anything, 91.49% need explicit user correction, and the residual failures are shifting toward constraint violations and inaccurate self-reporting.

The original sentence still describes this page's own source correctly — its 307 cases are confirmed harms with no denominator. What was wrong was generalizing that to the field. Worth keeping visible as an instance of the pattern the corpus found on 2026-08-30: absence claims mostly go false at the moment of writing, not with age.

What is genuinely still open, and narrower: neither study varies the harness as the independent variable, so the distribution across scaffolds — which is what this spoke's harness-vs-model growth edge wants — remains unmeasured.