ARC-AGI

ARC-AGI measures how well a model solves tasks it never saw in training, including ones people handle easily opus-5-arc-agi-3. It targets fluid reasoning rather than stored knowledge, which makes it the odd one out among the benchmarks in llm-benchmarks: GPQA, SWE-Bench and MMLU all reward what a model knows or can do in a familiar shape, while ARC-AGI is built so that familiarity is the thing being denied. It is run by ARC Prize.

Three generations

• ARC-AGI-1 and ARC-AGI-2 are static puzzle sets. Both are close to saturated at the frontier: Opus 5 posts 97.5% and 90.4%, matching prior top scores at slightly higher cost opus-5-arc-agi-3.
• ARC-AGI-3 is interactive. The model plays a game, and has to infer the rules of the environment, plan, and execute step by step rather than answer a question. That format is why scores collapse: the record before Opus 5 was 7.8% (GPT-5.6 Sol Max), and only six of 25 public demo environments have been solved at all opus-5-arc-agi-3.

The no-harness rule

Some systems have already beaten ARC-AGI-3 using a harness (scaffolding software around the model), but official scores count the language model’s performance alone. ARC Prize’s argument is that a future AGI shouldn’t need outside help to handle a new task opus-5-arc-agi-3. The practical consequence for reading the leaderboard: an official ARC-AGI-3 score is a floor on what the same model does inside an agent harness, and The Decoder notes Opus 5 would likely score higher run inside Claude Code. It also puts the benchmark deliberately at odds with how these models are actually sold, where the harness is the product (cross-spoke: agentic-tooling-wiki).

The contamination problem a novelty benchmark can’t escape

A benchmark whose whole claim is novelty is compromised the moment labs can aim at it. Opus 5 was built after ARC-AGI-3’s format went public, so targeted labeling and RL on the benchmark’s shape is available even without training on its tasks, and the evidence is mixed on whether that happened: the ~4× jump on ARC-AGI-3 does not reproduce on Guanghan Ning’s private Witness benchmark, where Opus 5 ties Fable 5 and Kimi K3 opus-5-arc-agi-3. Private or rotating evaluations are the usual answer, and they trade reproducibility for resistance to targeting.

Related

llm-benchmarks · arc-prize · opus-5-arc-agi-3 · claude-opus-5 · artificial-analysis · synthesis