MAST — "Why Do Multi-Agent LLM Systems Fail?"
Mert Cemri, Melissa Z. Pan, Shuyi Yang and colleagues. arXiv 2503.13657, March 2025; NeurIPS 2025. T1 — a peer-reviewed empirical taxonomy with a published dataset and inter-annotator agreement.
This is the paper agent-failure-localization said did not exist. That page has read, since 2026-08-06: "Nobody has published frequencies… the field still has no measurement of how agent failures actually distribute." MAST published exactly that seventeen months earlier.
The instrument
- 14 failure modes in 3 categories — system design issues, inter-agent misalignment, task verification.
- Built from 150 traces hand-annotated by experts at κ = 0.88 inter-annotator agreement, which is the number that makes a taxonomy a measurement rather than a vocabulary.
- Scaled by an LLM-as-judge pipeline validated against those human annotations, then run over MAST-Data: 1,600+ annotated traces across 7 multi-agent frameworks.
- Cut across models (GPT-4, Claude 3, Qwen2.5, CodeLlama) and tasks (coding, maths, general agent) — so the distribution can be read per model and per task, not only in aggregate.
The distribution
| Category | Share |
|---|---|
| System design issues | 44.2% |
| Inter-agent misalignment | 32.3% |
| Task verification | the remainder |
Within system design, the individual modes are published too: step repetition 15.7%, unawareness of termination conditions 12.4%, disobey task specification 11.8%, loss of conversation history 2.8%, disobey role specification 1.5%.
What the shape says
The largest single bucket is the system being built wrong, not the model being weak. Step repetition and not knowing when to stop are between them 28% of failures, and neither is a reasoning deficit — they are missing loop control. That lands directly on this spoke's standing finding that structure beats capability (agentic-tooling), and it lands from the failure side: the harness is where the value is and where the failures are.
Role and task specification barely register — 11.8% and 1.5%. Prompting the agents more carefully about who they are is not where the losses come from, which is the opposite of where most multi-agent effort goes.
Related
agent-failure-localization · coding-agent-misalignment · agent-orchestration · agent-evals · agentic-tooling · synthesis