More Skills, Worse Agents? Skill Shadowing (Song & Wei, Databricks, arXiv 2605.24050)
Hongwen Song and Song (Vinson) Wei of Databricks, submitted 2026-05-21, cs.SE. The first measurement this corpus holds of what happens when an agent sees every installed skill at once: all names and descriptions in context, no retrieval, the skill body loaded only when invoked. That is how agent-skills are installed in practice, and it is the case how-many-tools-should-an-agent-see never tested.
The setup
Two models, Claude Haiku 4.5 and Claude Sonnet 4.6, on SkillsBench (88 tasks). The authors kept only tasks where every skill in the task's own bundle gave at least 4% uplift, leaving 38 (task, model) pairs and 2,545 trajectories. Each task runs with its own skills only (the oracle library), then inside libraries of 52, 102 and 202 skills. The 202-skill library is every authored bundle merged and de-duplicated. The paper does not name the agent harness.
The numbers
Pass-rate drop against the oracle-only library, pooled across both models (95% CIs):
| Skills in context | Total drop | Share from shadowing | Share from context length |
|---|---|---|---|
| 52 | 0.08 [0.02, 0.15] | 0.03 (CI includes 0) | 0.06 (CI includes 0) |
| 102 | 0.14 [0.09, 0.19] | 0.08 [0.02, 0.15] | 0.06 (CI includes 0) |
| 202 | 0.21 [0.15, 0.27] | 0.14 [0.06, 0.26] | 0.07 (CI includes 0) |
So the drop is already measurable at 52 skills and reaches 21 points at 202. Shadowing is the authors' name for a distractor skill being picked in place of the right one, borrowed from variable shadowing. It explains up to 68% of the loss. The context-length share is the same size at every library size and never separates from zero.
The two models fail differently. Haiku's drop is larger (0.10 / 0.22 / 0.26) and mostly comes from abandonment: at 202 skills, 65.9% of its failures invoke no skill at all. Sonnet's drop is smaller (0.06 / 0.05 / 0.15) and comes from invoking the wrong skill.
What it settles, and what it does not
attention-is-all-you-have argued from the long-context literature that installed skills compete for "fewer than a hundred" reliable trigger slots. This paper measures the same thing directly and finds the decline starts earlier and is not a cliff: 8 points at 52, 14 at 102, 21 at 202. The mechanism is selection, not context length, which matches the Meta finding in how-many-tools-should-an-agent-see that the number of candidates shown is the variable. Here every installed skill is shown, so installed count and shown count are the same number.
Limits the authors state: two models from one provider, 202 skills at most, no mitigation tested (retrieval pre-filtering, description disambiguation and learned routing are proposed only). Absolute pass rates per library size are not reported, only the drops. Two internal inconsistencies in v1: the introduction says one distractor was invoked in "all 26 trajectories" where Table 3 lists 51, and Tables 4 and 5 give different CIs for the same per-model drops. Neither changes the pooled result.
Related
attention-is-all-you-have · how-many-tools-should-an-agent-see · agent-skills · skill-induced-failure · atskills-protocol · synthesis