SkillsBench

A benchmark for whether agent skills work — tasks paired with curated skills and deterministic programmatic verifiers, run under three settings: no skill, a curated skill, and a skill the agent writes for itself (Li, Liu, Chen et al., arXiv:2602.12670, submitted 13 February 2026).

Read from the paper 2026-09-05, and the inventory had moved. This page was written on 2026-08-15 out of agent-skills-can-be-harmful, which cited 84 tasks across 11 domains. The paper itself now says 87 tasks across 8 domains, and states the count as a current inventory rather than a fixed suite — so the two figures are the same benchmark at two moments, not a disagreement. Cite the version you read, and do not carry a task count across sources.

The headline the second-hand account did not carry

Under matched no-Skills and curated-Skills conditions over 18 model-harness configurations, the average pass rate goes 33.9% → 50.5% — +16.6 percentage points, a 25.5% normalized gain, with per-configuration gains from +4.1 to +25.7 pp. Two findings sit beside it:

• Skills an agent writes for itself yield "negligible or negative" gains. The value is in the curation, not in the format — which is the same shape self-improving-agents keeps running into.
• Focused skills of at most three modules beat larger or exhaustive bundles, and a smaller model with skills can match a larger model without them.

The result this spoke has been citing is the other half, from the failure study: curated skills raise the average and drop it on 16 tasks. Both are true of the same run, and together they are the argument — an average gain with a sixth of the suite going backwards is what motivated a failure taxonomy rather than another utility benchmark.

It is also the harness JetBrains borrowed to A/B ponytail — 80 paired tasks, auto-graded 0–1 with partial credit (jetbrains-ponytail-tested) — so the same task set now backs both the largest independent skill evaluation in this corpus and the largest failure taxonomy.

What it is good for, and where it stops

Deterministic verifiers plus executable environments make pass/fail and cost comparable across runs, which is the property both later studies needed. The limit is the comparison space: two with-skill conditions per task means a failing run usually has no successful alternative skill to be compared against, so the benchmark can show that a skill hurt without localizing what in it did. The follow-up study widened this by retrieving semantically matched public skills — up to six variants per task, and 504 with-skill/no-skill plus 2,520 cross-skill potential pairs where the original design had 168 and 168.

Its tasks span software engineering, healthcare, manufacturing, cybersecurity, natural science, finance and mathematics — the breadth is why skill-induced-failure's taxonomy is not purely a coding-agent artifact, though half its cases still come from the repository-based swe-skills-bench.

Related

swe-skills-bench · agent-skills-can-be-harmful · skill-induced-failure · jetbrains-ponytail-tested · agent-evals · swe-bench · agent-skills