Agent failure localization
Deciding which component of an agent system caused a failure, so you know which repair to attempt. Distinct from failure detection (something went wrong) and failure classification (what kind of wrong): localization asks whose fault, because the answer selects between four unrelated interventions — model post-training, harness engineering, environment redesign, or benchmark repair model-or-harness-taxonomy.
Why outcome labels are not enough
An agent ignores an instruction given earlier in a long session. Either the harness’s context compaction removed it, or it was still present and the model didn’t follow it. The trace shows the same behaviour; the fix is in different codebases model-or-harness-taxonomy. This is the repair-assignment problem, and it is the reason a score on a benchmark — the thing agent-evals produces — does not tell a team what to change next.
The mechanism: interaction as the unit
Scale AI’s taxonomy makes the interaction between two components the
thing being labelled, not the component itself. Each failure gets an edge (which two components
were interacting) plus a fault side (which of the two is responsible), written
COMP1—COMP2 · fault:SIDE. A tool wrapper that swallows an error and a model that ignores a returned
error sit on the same edge with opposite fault sides, and want opposite fixes.
Two design choices carry most of the weight:
- A fixed root-cause rule. Trajectories cascade, so without a rule two annotators label the same trace differently. The rule: trace backward from the observed failure to the earliest failure from which execution does not recover; everything after is a symptom.
- Roles instead of new components for multi-agent cases — a peer or subagent is another model,
so it sits on a
MODEL—MODELedge with a role, and the vocabulary doesn’t grow to cover orchestration.
Where the fault lands, and the bias in that
Of 41 modes, 36 are model-side — but by construction: fault is model-side whenever a stronger model could have avoided or recovered from the failure. The five non-model modes are the load-bearing ones, since they mark failures that no model upgrade fixes model-or-harness-taxonomy. The same bias reappears in practice: LLM judges applying the taxonomy over-attribute to the model when a task fails.
This is the mirror image of the spoke’s own standing claim. agentic-coding-harness argues value has moved from the model to the scaffolding around it; a taxonomy built to locate blame nevertheless defaults blame to the model. Both can be true — structure decides how work gets done, while the capability ceiling stays the model’s. The spoke has no measurement that settles it either way.
Relation to the spoke’s other failure lists
The spoke already holds failure inventories, and they are doing a different job:
- agent-anti-patterns lists nine build-time mistakes (too many tools, no memory architecture, no observability). Prescriptive, before the fact.
- context-rot and cca-f-lost-in-the-middle describe one failure mechanism in depth.
- Localization is post-hoc and diagnostic: it takes a trace that already failed and assigns it.
The practical link is observability. Assigning fault to an edge requires a record of what crossed that edge, which is the case agent-evals already makes for tracing — the score tells you that you regressed, the trace plus a localization label tells you where to look.
Open
Nobody has published frequencies. The taxonomy is explicitly descriptive and its 40 worked examples were chosen for coverage, so the field still has no measurement of how agent failures actually distribute across model, harness and environment — which is the number the spoke’s harness-vs-model growth edge ultimately wants.