CodeRabbit

A vendor of AI code review: the product reads a pull request and posts review comments, which makes frontier LLMs its raw material rather than its subject. It publishes per-model evaluations on its blog as each new model lands; coderabbit-fable-5-1-review is the first this spoke holds.

That position is what makes its numbers useful and what limits them. CodeRabbit runs the same harness across models on a task shape the labs' own benchmarks do not isolate (reviewing a diff for defects someone already catalogued), and it reports recall, precision, comment volume and latency together rather than a single score, the multi-axis shape llm-benchmarks argues for. It is also an interested party: its measured precision figures of 31.6%–39.3% are the argument for the filtering layer it sells.

Neither the task set nor the scoring rubric is published, so nothing here is reproducible.

Related

coderabbit-fable-5-1-review · claude-fable-5-1 · llm-benchmarks