LLM benchmarks & leaderboards

How models are ranked across the llm-provider landscape. The 2026 best practice (llm-leaderboard-stats) is a composite, multi-axis score, not a single number.

The axes

Capability benchmarks — GPQA Diamond, SWE-Bench Verified, MMLU, AIME, HumanEval, coding-arena.
Speed — output throughput (tokens/sec), time-to-first-token.
Price — per-token cost (llm-api-pricing).
Context window and agentic/coding performance.
Aggregators referenced across sources: Artificial Analysis Index, llm-stats.com.

Anatomy of a composite — the Artificial Analysis Intelligence Index

The clearest worked example of "composite, not a single number." As of v4.1 (current, 2026-06) the artificial-analysis Intelligence Index is a weighted average of 9 evals across four categories, and the weighting is itself a thesis about what matters in 2026 artificial-analysis:

Agents — 34% (the biggest slice): GDPval-AA v2 (220 agentic file-output tasks), τ³-Banking (97 dual-control agent-user tasks).
Coding — 24%: Terminal-Bench v2.1 (89 terminal tasks), SciCode (288 code subproblems).
Scientific reasoning — 24%: Humanity's Last Exam (2,158 open questions), GPQA Diamond (198 MC), CritPt (70 physics challenges).
General — 18%: AA-LCR (100 long-context-reasoning questions), AA-Omniscience (6,000 knowledge questions, split 8% accuracy + 4% non-hallucination rate).

Two things follow. Agents+coding = 58% of the score, so the index now rewards doing tasks over recalling facts; and the disclosed ±1% 95% confidence interval (from >10 repeats) is what lets it function as the reproducible yardstick the synthesis leans on. The weighting is a defensible editorial choice, not ground truth — the standing caveat on every leaderboard.

2026 snapshot (volatile — churns weekly)

Reasoning leaders are proprietary — Anthropic (Claude Mythos Preview, claude-opus-4-8 #2) and OpenAI (GPT-5.5 #3); Alibaba's Qwen3.7 Max is the cheapest top-10 ($1.53/M); xAI's Grok 4 Fast has the largest context (2.0M); Mercury 2 is fastest (784 tok/s). Among open weights, deepseek V4 Pro tops the Artificial Analysis Index; Qwen3.6 hits 77.2% SWE-Bench.

The other kind — novelty benchmarks

Everything above measures a model on tasks whose shape is known in advance, which is what makes them reproducible and what makes them gameable. ARC-AGI is built the other way: it scores tasks the model never saw in training, and its current version (ARC-AGI-3) makes the model infer the rules of an interactive game rather than answer a question opus-5-arc-agi-3. Two rules follow from that design and both are unusual here.

Model alone, no harness. ARC Prize excludes scaffolding software from official scores, so an ARC-AGI-3 number is a floor on what the same model does inside an agent. Every index in this page's table measures the opposite: whatever the vendor ships.
Scores are far lower and move in jumps. The ARC-AGI-3 record went 7.8% → 30.2% in one model generation, where composite indices creep. A benchmark nobody has saturated discriminates; one at 90%+ mostly measures the noise floor, which is where ARC-AGI-1 and -2 now sit.

The catch is that a novelty benchmark stops measuring novelty once labs can aim at it, and the Opus 5 result is the live test case: a ~4× jump on ARC-AGI-3 that doesn't reproduce on a private benchmark opus-5-arc-agi-3.

A vendor harness measuring one task, and what it exposes (2026-09-01)

The indices above score a model; coderabbit-fable-5-1-review scores a job — 45 code-review tasks against 105 catalogued defects, reported as recall and precision and comment volume and latency. Three things show up that a composite score hides.

Nobody wins on both axes. GPT-5.6 Sol has the best recall (69.7%) and the worst precision (31.6%); Opus 5 the reverse (55.2% / 39.3%); Fable 5.1 sits between them (61.0% / 37.3%). A single ranking would have to pick which error matters, and the honest answer depends on who reads the output.

The floor is low everywhere. Precision runs 31.6%–39.3% across all four models, so on this task most of what any of them says is not marked valid. That is not visible in any capability index in this page, which report the share of problems solved and never the share of output that was wrong.

Raising effort lowered the score. Fable 5.1 at high reasoning was slower (21:36 vs 18:38) and found less (57.1% vs 61.0% recall). Effort is treated across this spoke as a cost/quality dial; here it was neither. One harness, unpublished, no repeat runs — recorded as a thing to watch, not a result.

The trade for all of it is reproducibility: the task set and rubric are not published, and the harness belongs to a company selling the filtering layer that a 37% precision figure argues for (coderabbit).

The caution

Single-benchmark rankings invite gaming and hype; the open-source-llms-2026 source explicitly argues practical fit (task performance, license, hardware, cost, speed) beats leaderboard rank. Treat every ranking as a dated snapshot, and prefer composite/multi-axis views.

llm-leaderboard-stats · llm-provider · llm-api-pricing · open-weight-models · claude-opus-4-8 · arc-agi · arc-prize · opus-5-arc-agi-3 · coderabbit-fable-5-1-review