SWE-Skills-Bench

Do agent skills actually help in real-world software engineering? — 490 repository-based task instances with deterministic tests, run with and without a curated skill (Han et al., arXiv:2603.15401, 2026). Known here through agent-skills-can-be-harmful, which uses it as its second study subject; the original paper has not been read into this spoke.

Its reported answer is the sceptical one: limited average improvement, many skills with no pass-rate gain at all, and token overhead reaching 451% on individual cases. That number is the reason the follow-up study treats cost as a first-class failure class rather than a footnote — a skill that quadruples the bill on a task it also solves never shows up in a pass-rate table.

Repository-based instances make it the swe-bench-shaped half of the skill-failure evidence: real codebases, real test suites, and therefore the file paths, package layouts and dependency states that turn out to produce a third of all confirmed functional failures (skill-induced-failure). Its narrower design — one with-skill condition per task, no cross-skill comparison at all — is what the failure study had to widen, from 490 potential pairs to 17,640 by retrieving matched public skills.

Related

skillsbench · agent-skills-can-be-harmful · skill-induced-failure · swe-bench · agent-evals · agent-skills