SWE-bench

Jimenez, Yang, Wettig, Yao, Pei, Press and Narasimhan (Princeton), SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, ICLR 2024. T1. The benchmark this spoke has argued about in seven pages without holding it.

2,294 task instances built from real issues and their merged pull requests across 12 popular Python repositories. A model gets the codebase and the issue text and must produce a patch; the patch is judged by the repository's own tests. The paper's own emphasis is on scope: tasks require "understanding and coordinating changes across multiple functions, classes, and even files simultaneously."

At publication, the best model resolved 1.96% (Claude 2). That number is the thing to remember whenever a current score is quoted — the benchmark was designed as a wall, and what followed was less a story about models than about harnesses, which is this spoke's whole subject. A model alone scored ~2%; the same class of model inside a scaffold with retrieval, editing tools and test feedback scores an order of magnitude higher.

What it does and does not measure

Tests decide. That makes grading objective and cheap, and it also fixes what "resolved" means: patches that pass the repository's tests, on Python, on issues whose fix was accepted upstream. It says nothing about design quality, about work with no test to encode it, or about the long-running multi-step behaviour ddr-bench and harness-bench examine. Its known weaknesses — solution leakage from issue text, environment fragility — are why the filtered SWE-bench Verified subset exists; this spoke holds neither a source on Verified nor one measuring contamination.

Cross-link: claw-swe-bench is a vendor score against this benchmark, and should be read against the paper rather than against other vendors' claims.

How much of a current score is memory (added 2026-08-22)

swe-bench-illusion (Liang, Garg & Zilouchian Moghaddam, 2025) measured what ten models know about these repositories without seeing them. Given only the issue text, they name the buggy file at 60–76% on Verified — 763 files per repository on average — and below 53% on repositories outside the benchmark; verbatim 5-gram recall of the fixed function runs 35% here against 18% elsewhere. The task set has been public since 2023, and the ordering of that drop is what contamination looks like. The harness arc on this page still holds; the resolve rates on top of it carry an unknown memorisation component, so treat small differences between reported scores as noise.