Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents
arXiv:2608.11888v1 [cs.AI], 12 Aug 2026. Gen Dong (Huazhong University of Science and Technology, work done during an internship at Microsoft Research), Yanjie Gao (microsoft Research, corresponding), Liqun Li (Microsoft), Tianyin Xu (University of Illinois Urbana-Champaign), Yu Hua (HUST), Fan Yang (Microsoft Research). T1 — preprint with primary data: 307 hand-labelled failure cases from controlled paired runs.
This is the study the spoke has been asking for from the other direction. cc-thinking-skills ran an eval over its own 39-skill pack and found no proven gain; jetbrains-ponytail-tested A/B'd one skill and found a third of the advertised effect. Neither could say why. This paper takes the two public skill benchmarks, builds a contrastive design that pins a failure on the loaded skill, and classifies 307 of them. The headline is in the title: a skill that is topically right for the task is the normal way this goes wrong.
The method: a reference run as pseudo-oracle
Borrowed from differential testing. Two runs of the same task hold everything fixed — instruction, repository or container state, data, verifier, harness, model — and vary only the skill setup. The audited run is the target; the reference is either a no-skill run or a run with a different, semantically matched skill.
- Functional failure: target FAILs the verifier, reference PASSes.
- Efficiency regression: both PASS, but the target costs substantially more. Formally, with
r_tokandr_timeas target-to-reference ratios,min(r_tok, r_time) > 1.0andmax(r_tok, r_time) > T, at a primary threshold T = 2.0. Requiring both ratios above 1.0 throws out token-for-time trades; requiring one to more than double keeps run-to-run noise out.
The reference run is not a ground-truth solution. It only establishes that this task, under this verifier, was solvable — or solvable cheaper — so the difference can be charged to the skill rather than to the model, the task, or ordinary variance. That is the move agent-failure-localization describes as repair assignment, applied to a component nobody had isolated before: the skill file.
Setup
Subjects are skillsbench (84 verifier-graded tasks, 11 domains) and swe-skills-bench (490
repository tasks with deterministic tests). Both ship a narrow comparison space — no-skill versus one
or two curated skills — which is enough to show a skill helps or hurts and not enough to localize why.
So the authors augmented the skill pool from public skill marketplaces (smithery.ai, skillsmp.com):
for each curated skill they queried by name and description, embedded the metadata with
all-MiniLM-L6-v2, and kept up to 5 candidates at cosine similarity ≥ 0.7. Up to six skill variants
per task, and the comparison space grows from 826 to 20,664 potential pairs — roughly 25×.
Retrieved skills were offered through the normal skill-loading interface, not force-fed into context: the agent could decline to load one. Runs used OpenCode 1.15.1 with Claude Opus 4.6.
Executed evaluations produced 665 labelled candidates; refinement dropped ambiguous cases, verifier-narrow false positives and duplicate same-task/same-skill effects, leaving 307 confirmed cases — 125 functional failures and 182 efficiency regressions — each given one root-cause label by hand and finalized by group consensus.
What breaks the task (125 functional failures)
| Category | Count | Share |
|---|---|---|
| Task-Implementation Fault — wrong or missing required element | 86 | 68.8% |
| ├ Incorrect Required-Element Fill | 46 | 36.8% |
| ├ Required-Element Omission | 36 | 28.8% |
| └ Obstructive Workflow Guidance | 4 | 3.2% |
| Artifact Misplacement — right artifact, wrong path | 24 | 19.2% |
| Environment Mismatch | 13 | 10.4% |
| ├ Environment-State Mismatch | 8 | 6.4% |
| └ Broken Dependency or Runtime | 5 | 4.0% |
| Applicability Mismatch — skill should not have loaded | 2 | 1.6% |
The distribution is the finding. Loading the wrong skill accounts for 2 cases out of 125. What
actually happens is that an on-topic skill's reusable defaults, examples and templates get treated as
task-specific requirements. A spreadsheet skill computes (Exports - Imports) / GDP and the verifier
wants percent, so the answer is 100× too small. A RAG skill's example exposes chunk size, overlap and
top-k as CLI flags but never shows model parameters, and the agent ships exactly the example's
surface, omitting the configurable model_name the task required.
The other two categories sit on what the authors call execution-surface boundaries — 37 of 125 cases
where the skill did not get the content wrong, it changed where the work landed or under which
runtime state it was checked. An openpyxl case: the skill's dependency advice led the agent to
pip install a modern version, cd /tmp, and strip the repo path from sys.path. Its self-check
passed against the external install while the verifier imported from the repository package. In a
LangChain task the agent reasoned that "the package is langchain_classic" and wrote there instead of
the task-specified libs/langchain/langchain/ — plausible components, failed file-existence tests.
What makes it expensive (182 efficiency regressions)
| Category | Count | Share |
|---|---|---|
| Excessive Procedure | 114 | 62.6% |
| ├ Excessive Verification | 67 | 36.8% |
| ├ Heavy Implementation Pipeline | 30 | 16.5% |
| └ Excessive Exploration | 17 | 9.3% |
| Context Bloat | 46 | 25.3% |
| ├ Skill-Body Context Bloat | 43 | 23.6% |
| └ Supplementary-Material Bloat | 3 | 1.6% |
| Dependency Resolution | 22 | 12.1% |
Two results here matter for the progressive-disclosure argument. First, cost is mostly not prompt length: nearly two-thirds of regressions come from the skill adding steps, not tokens per call. The single largest bucket in the whole table is excessive verification — 67 cases where the skill kept the agent testing, rebuilding, debugging and walking checklists after the artifact was already correct. Second, when cost is context, it is 43 of 46 cases the mandatory skill body, against 3 for the supplementary files. The always-loaded text has a far wider cost surface than the optional references, because the body sits in every model call while a reference costs only when read.
SKILLTRIAGE
The taxonomy is also run as an attribution procedure. SKILLTRIAGE normalizes a confirmed pair into a task view plus two run views, extracts differential evidence — five signals for functional failures (environment change, whether it was skill-prescribed, task path versus written path, whether the artifact was produced, construction difference), phase- and tag-level cost deltas for regressions — and then picks the label that best explains both the outcome and the trajectory divergence. GPT-5.5, three runs, 2-of-3 majority vote.
| exact subcategory | high-level category | largest-category subcat. | |
|---|---|---|---|
| Functional failures | 111/125 (88.8%) | 117/125 (93.6%) | 76/86 (88.4%) |
| Efficiency regressions | 132/182 (72.5%) | 145/182 (79.7%) | 78/114 (68.4%) |
Residual errors cluster on taxonomy boundaries rather than scattering: incorrect-fill versus omission, environment mismatch versus wrong location, and — the harder half — dependency repair nested inside a verification loop, where one expensive trajectory offers several plausible cost surfaces. The authors' own conclusion is that a triage tool should return the cited skill section and trajectory step, not just a label. Note what it is not: it triages pairs already confirmed as skill-induced. It does not find them.
What the paper recommends
- Skill-task compatibility checks before and during execution — compare required fields, APIs, paths, output formats and environment constraints against what the skill's defaults and templates would impose, and warn on conflict.
- Cost-aware packaging: short always-loaded bodies, examples and checklists behind lazy-loading triggers, and an estimate of induced work — not just added tokens — before a skill is selected.
- Budget-aware execution policies that scale verification depth to task uncertainty, change size and remaining budget instead of running the skill's full checklist every time.
- For platforms: treat task-specified paths, integration points and package state as guarded constraints that skill-guided changes must not silently cross.
Limits worth carrying
These are counts of failures, not a failure rate. The 307 cases are the confirmed harmful pairs; the paper does not report how many executed pairs were fine, so nothing here says how often a skill hurts. It says what harm looks like when it happens. The prior benchmarks are the ones holding rates — skillsbench saw pass-rate drops on 16 of 84 tasks, swe-skills-bench token overhead up to 451%.
The authors flag the rest themselves. Labelling required manual judgment on questions that stay subjective — whether a verifier is narrower than its task, whether a cost delta is just variance — mitigated by dropping ambiguous cases and by consensus. And the whole study runs on one harness and one model (OpenCode + Claude Opus 4.6) over two benchmarks, so the task distribution and skill ecosystem could carry some of the shape. The harness-varies-the-result finding this spoke already holds is a direct reason to expect that.
Related
skill-induced-failure · skilltriage · skillsbench · swe-skills-bench · agent-skills · agentskills-spec · agent-failure-localization · agent-evals · context-rot · verification-loops-skills · cc-thinking-skills · jetbrains-ponytail-tested