Skill-induced failure

A task that an agent would have completed — or completed cheaper — without the skill it loaded. The name and the taxonomy come from agent-skills-can-be-harmful, which confirmed 307 of them against paired reference runs. Two classes:

• Functional failure — the run with the skill fails the verifier; a no-skill or matched-skill run of the same task passes.
• Efficiency regression — both runs pass, but the skill's run costs substantially more in tokens and wall time (the paper's bar: both ratios above 1.0, at least one above 2.0).

The class only exists because of the comparison. A failed run with a skill loaded proves nothing on its own — it could be the model, the task, the verifier, or variance. Pinning it on the skill needs a run that solved the same task without it, which is why the term arrived with a contrastive method attached and not before.

The counter-intuitive part

The obvious hypothesis is that harm comes from loading an irrelevant skill. It does not. Applicability Mismatch — the skill should not have been loaded at all — is 2 of 125 functional failures (1.6%). The other 123 come from skills that were topically right and operationally wrong for the concrete task.

The dominant mechanism is what the paper calls Task-Implementation Fault (86 cases, 68.8%): the agent over-trusts a matching skill and treats its reusable defaults, examples and templates as the task's requirements. It fills a required element with the skill's method instead of the task's (46 cases), or leaves an element out because the skill's example did not show it (36 cases). This is not the agent ignoring instructions. It is the agent following the wrong instructions faithfully.

A second cluster (37 cases) never touches the content at all. Artifact Misplacement (24) writes a correct artifact at a path the skill's layout conventions suggested rather than the one the task specified; Environment Mismatch (13) changes package resolution, working directory or installed versions so the agent's self-check and the verifier no longer see the same world. Both are cases where the skill moved a verifier-visible surface out from under an otherwise good run.

Cost has two sources, and the smaller one gets the attention

Efficiency regressions split by where the extra cost comes from. Context Bloat (46 cases) makes each model call dearer while the trajectory stays roughly the same shape. Excessive Procedure (114 cases) changes the trajectory — extra exploration, heavier construction pipelines, and above all extra verification, the single biggest bucket at 67 cases. A third category, Dependency Resolution (22), is the tax on repairing a fragile runtime the skill asked for, in runs that eventually pass.

So the intuitive worry — the skill makes the prompt longer — is about a quarter of the problem, and within that quarter it is the mandatory body (43 of 46) rather than the optional references (3). The skill's real expense is the work it makes the agent do.

This is the first measured limit on the progressive-disclosure bargain that is not about how many skills fit. agentic-awesome-skills found the aggregate ceiling — at 2,003 skills the index itself stops fitting. This finds a per-skill one: the cheap-frontmatter argument prices the L1/L2 text and prices nothing about the procedure the L2 text prescribes. A short skill that mandates a full test suite after every edit is cheap to load and expensive to obey.

Why a "skills eval" doesn't catch it

An eval scores outcomes on a dataset. The two failure classes here are invisible to that in different ways: an efficiency regression passes, so a pass-rate eval scores it green; and a functional failure shows up as one point of a pass rate that the average across the suite can absorb — skillsbench reported average gains while dropping pass rate on 16 of 84 tasks. Attribution is a different operation from scoring, which is the point agent-failure-localization makes about components generally, arriving here for a specific one.

Repairs the evidence points at

• Mark what is mandatory. The largest category exists because examples, defaults and templates read as requirements. Separating them is an authoring convention, not a platform feature.
• Keep the always-loaded body short and lazy-load the rest — the 43-versus-3 split says the standard's own L2/L3 boundary is the lever, and that authors put too much above it.
• Condition verification on the change, rather than shipping an exhaustive checklist that becomes mandatory work on every task. Cf. verification-loops-skills, which is the practice this finding taxes: the loop is worth encoding, and a loop with no stopping rule is the top cost bucket in the corpus.
• Guard the task-specified path and the environment state against skill-suggested substitution.

Related

agent-skills-can-be-harmful · skilltriage · agent-skills · agentskills-spec · agent-failure-localization · agent-evals · agent-anti-patterns · context-rot · skillsbench · swe-skills-bench