The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason (2025)

Shanchao Liang, Spandan Garg and Roshanak Zilouchian Moghaddam. arXiv 2506.12286, submitted 14 June 2025, revised 1 December 2025. T1 — a contamination study measuring published models against a published benchmark, with an instrument the authors define and defend.

The contamination-and-saturation source growth edge 7 has been asking for since swe-bench landed here.

The instrument: ask for something a solver would not know

The clever part is what they measure. Rather than re-running agents, they give models two diagnostic tasks that a genuine problem-solver could not do well:

1. File path identification — name the buggy file from the issue description alone, with no access to the repository structure.
2. Ground-truth function reproduction — reproduce the fixed function given only the current file context and the issue.

Neither is solvable from reasoning about a codebase you cannot see. Doing well means the artefact is already in the weights.

Ten models: GPT-4o (May and August 2024 snapshots), GPT-4.1, o3, o3-mini, o4-mini, Claude 3.5 Sonnet, Claude 3.7 Sonnet, Claude 4.0 Sonnet, Claude 4.0 Opus.

The result

File-path identification, with no repository access:

Seventy-six percent, from prose, on repositories averaging 763.5 files. And the drop is ordered the way contamination predicts: highest on the most-published subset, lower on the same benchmark's unverified tail, lowest on repositories the benchmark never covered.

Function reproduction agrees: up to 35% consecutive 5-gram accuracy on SWE-Bench Verified and Full, against at most 18% on comparable tasks elsewhere. Verbatim recall of the fix, not a plausible reconstruction of it.

One number stands on its own: Claude Opus scored 92.11% when the ground-truth path was mentioned somewhere in the issue text and 67.41% when it was not. Even the "hard" half is far above what blind guessing over 763 files could produce.

What it does and does not license saying

It does not show that reported SWE-bench resolve rates are fake. A model can have memorised where a bug lives and still need a harness to fix it — and the harness is what this spoke measures. What it shows is that part of the score is retrieval from training data, that the part grows with how heavily a task set has been published, and therefore that a cross-model comparison on SWE-Bench Verified is partly a comparison of training-set overlap.

The authors' own caution is on the record: the drop is "partly compounded by confounding variables, such as the significantly shorter issue descriptions in RefactorBench" — 14.6 tokens against SWE-Bench Verified's 451.2 — and the mentioned-path split uses "simple heuristics." The pandas/PyTorch comparison is the cleaner half of the evidence, since those are large, popular Python repositories with issue text of the ordinary kind.

Consequences for pages in this spoke

• Every current score quoted here against swe-bench — and the 1.96%-to-70% arc that makes the harness argument — sits on a benchmark whose task set has been in training corpora for two years. The arc is still real; its endpoint is now an upper bound with an unknown memorisation component.
• It sharpens what agent-storage-footprint found from the other direction: 108 SWE-bench Verified submissions whose trajectory volume spans three orders of magnitude with no correlation to resolve rate. Two independent results, both saying the headline number carries less information than its precision suggests.
• The comparison set includes SWE-Bench-C# (40 instances), so the drop is not only across repositories but across a language boundary. The paper does not break out a per-language figure this page can quote, and none is asserted here.

Related

swe-bench · harness-bench · agent-storage-footprint · tau-bench · webarena · claw-swe-bench · agent-framework