SWE-bench

Jimenez, Yang, Wettig, Yao, Pei, Press and Narasimhan (Princeton), SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, ICLR 2024. T1. The benchmark this spoke has argued about in seven pages without holding it.

2,294 task instances built from real issues and their merged pull requests across 12 popular Python repositories. A model gets the codebase and the issue text and must produce a patch; the patch is judged by the repository’s own tests. The paper’s own emphasis is on scope: tasks require “understanding and coordinating changes across multiple functions, classes, and even files simultaneously.”

At publication, the best model resolved 1.96% (Claude 2). That number is the thing to remember whenever a current score is quoted — the benchmark was designed as a wall, and what followed was less a story about models than about harnesses, which is this spoke’s whole subject. A model alone scored ~2%; the same class of model inside a scaffold with retrieval, editing tools and test feedback scores an order of magnitude higher.

What it does and does not measure

Tests decide. That makes grading objective and cheap, and it also fixes what “resolved” means: patches that pass the repository’s tests, on Python, on issues whose fix was accepted upstream. It says nothing about design quality, about work with no test to encode it, or about the long-running multi-step behaviour ddr-bench and harness-bench examine. Its known weaknesses — solution leakage from issue text, environment fragility — are why the filtered SWE-bench Verified subset exists; this spoke holds neither a source on Verified nor one measuring contamination.

Cross-link: claw-swe-bench is a vendor score against this benchmark, and should be read against the paper rather than against other vendors’ claims.