CodeRabbit's Fable 5.1 code-review benchmark
CodeRabbit (Juan Pablo Flores and Gowtham Kishore Vijay, 1 Sep 2026) ran Claude Fable 5.1 through its own code-review harness and published the numbers beside Fable 5, Opus 5 and OpenAI's GPT-5.6 Sol: 45 review tasks scored against 105 known issue points, at two reasoning settings.
The numbers
| Recall | Precision | Comments | Nitpicks | Latency | |
|---|---|---|---|---|---|
| Fable 5.1 (low reasoning) | 61.0% | 37.3% | 166 | 79 | 18:38 |
| Fable 5.1 (high reasoning) | 57.1% | — | — | — | 21:36 |
| Fable 5 | 61.9% | 32.8% | 253 | 265 | 12:32 |
| Opus 5 | 55.2% | 39.3% | 166 | — | — |
| GPT-5.6 Sol | 69.7% | 31.6% | 231 | — | — |
Three findings
More reasoning made it worse. High reasoning took 21:36 against 18:38 and found less: 57.1% recall against 61.0%. The authors say the extra time "did not improve the overall result." Every assumption in this spoke about the effort knob has been that it trades cost for quality; here it bought neither. One task family, one harness, and no repeat runs are reported, so this is a data point rather than a result about effort in general.
The version upgrade bought quiet, not sight. Against Fable 5, Fable 5.1 gained 4.5 points of precision and lost 1 point of recall, cut total comments by 34% and nitpicks by 70% (265 to 79), and took 49% longer. The model found the same share of real problems and said much less about everything else.
No model is good at this, and none wins on both axes. Precision runs 31.6%–39.3% across all four, so on this harness most comments any of them produce are not marked valid. The authors' own line is that "37.3% precision means most final comments were not marked valid." The ordering also flips by metric: GPT-5.6 Sol has the best recall and the worst precision, Opus 5 the reverse, with Fable 5.1 between them. Picking a "best" model here means picking which error you would rather have.
The authors' conclusion is that Fable 5.1 is "a specialist reviewer, not an automatic replacement," to be used "selectively" on "high-complexity changes," with a validation layer over it.
What to hold against it
A vendor benchmark, and the vendor sells the validation layer its conclusion recommends: CodeRabbit's product is exactly the filter that a 37% precision number argues for. That is not a reason to discount the figures, which are specific and self-consistent; it is a reason to want the harness. The 45 tasks and 105 issue points are not published, no repeat runs or confidence intervals are given, "marked valid" is not defined, and Opus 5's and GPT-5.6 Sol's latency and nitpick counts are missing. T2 because it measures rather than relays, not because it is independent.
Related
claude-fable-5-1 · claude-fable-5 · claude-opus-5 · coderabbit · llm-benchmarks · llm-api-pricing