Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures

Harsh Raj, Vipul Gupta, Anas Mahmoud, Razvan-Gabriel Dumitru, Darvin Yi, Aakash Sabharwal and Yunzhong He, Scale AI (arXiv:2607.28802v1, cs.AI, 2026-07-30). A scheme for saying which component of an agent system broke, and therefore which repair to attempt. It is the first source in this spoke that takes the model-versus-harness question the evals corner keeps circling and gives it a working vocabulary — while being explicit that it measures nothing.

The problem it names: repair assignment

Evaluations score outcomes, and an outcome label collapses causes. The paper’s own example is a Claude Code session where the agent ignores an earlier user instruction: the harness’s context compaction may have deleted it, or the instruction may still have been there and the model failed to follow it. “The observed behavior is the same, but the first case requires a harness-level fix, whereas the second requires a model-level intervention.” Same visible failure, different repair — post-training, harness engineering, environment redesign, or benchmark repair. Existing taxonomies don’t help because they are benchmark-specific or organize by which internal module was affected, neither of which says who is at fault.

The mechanism: edge plus fault side

An agent is modelled as interacting components, and the unit of analysis is the interaction, not the component. Nine components in three families:

• User — owner (gives the task, defines success), grader (evaluates, usually invisible to the agent), third party (an actor met during execution not acting for the owner).
• Harness — context, memory, tool, and another model in the role of peer or subagent.
• Environment — local env. (shell, filesystem, runtimes) and external env. (remote services, APIs, provider infrastructure).

A failure is written COMP1—COMP2 · fault:SIDE. The worked case: an agent reports a tool call succeeded when it failed. If the tool wrapper swallowed the error so the model never saw it, that’s TOOL—MODEL·fault:TOOL; if the wrapper returned the error and the model ignored it, that’s TOOL—MODEL·fault:MODEL. Same edge, different repair. This is the whole idea, and the concept page is agent-failure-localization.

Multi-agent failures need no new components — the other endpoint is still a model, so peer and subagent are roles on a MODEL—MODEL edge rather than component types. That is a cleaner move than the orchestration literature usually makes.

The tie-break rule. One trajectory contains cascading errors, so the paper fixes which one gets labelled: start at the observed system-level failure, trace the causal chain backward, and label the earliest failure from which execution does not recover — later errors are consequences.

The 41 modes

Grouped by the component the model is interacting with. A sample rather than the full list:

36 of the 41 are model-side, five are not — and the paper says plainly why that number is inflated: the attribution rule assigns fault to the model whenever “a more capable model could have prevented it or recovered from it” under the same conditions. So the imbalance is a property of the rule, not a measurement of where agents actually break. The five non-model modes carry the weight: those are the failures model improvement alone cannot fix.

Failures with safety or security impact get a second annotation drawn from the OWASP Top 10 for LLM and Agentic Applications, alongside the taxonomy label.

The validation: agents judging the labels

The authors froze the taxonomy, annotated 40 worked examples from public benchmarks, model system cards, published reports and logged trajectories, then asked whether independent judges recover the same labels from the same definitions. Four frontier models ran as separate judges — GPT-5.5 at xhigh reasoning effort, and Claude Opus 4.6/4.7/4.8 at max — each built as an agent on the Claude Agent SDK, given only a reference to the original source and made to reconstruct the evidence itself (evidence reconstruction → classification → reflection against disambiguation rules).

Judge-to-judge agreement is about as strong as judge-to-human (highest pairwise κ = 0.84, Opus 4.6 vs 4.8), which is the actual argument: the categories carry shared structure rather than one annotator’s preferences. Finer-grained failure-mode labels are weaker — GPT-5.5 reaches 0.72 accuracy there. A selective-voting ensemble that abstains unless k judges agree reaches 0.83 category precision at 90% coverage, buying precision with coverage.

The judges’ characteristic mistake lands squarely on this spoke’s thesis: when a task fails they tend to blame the model, even when the fault lies elsewhere. The same bias the taxonomy was built to correct shows up in the tool built to apply it.

What it does and does not settle

The paper states its own limits: “the taxonomy is descriptive rather than quantitative: it organizes failures and assigns responsibility but does not estimate their relative frequency.” The 40 examples were picked to span edges, and the authors warn them off explicitly — the set “should not be used to estimate the prevalence of individual failure modes.” Labels also depend on available evidence, and system cards often omit what’s needed to pin a unique root cause. On deployment, they concede judge accuracy is too low for production use, especially at failure-mode granularity.

So it does not close the spoke’s harness-vs-model growth edge, which asks for a controlled comparison varying model and harness in the same design. What it supplies is the thing that edge has been missing upstream of measurement: a principled, reproducible way to assign a given failure to one side or the other. You cannot count what you cannot classify.

T1 — first-party lab research with a frozen protocol, published agreement statistics, and stated limitations. It is a preprint, not peer-reviewed, and the validation set is 40 examples labelled by the taxonomy’s own authors.