How Many Tools Should an LLM Agent See? (Repantis et al., Meta, 2026)

Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh and Joey Blackwell II, Meta Platforms, arXiv:2605.24660, 13 pages — read from the PDF. Found by the research pass hunting the installed-skill count edge that attention-is-all-you-have opened and declined to measure.

The paper's move is to make shortlist depth the object of evaluation: before an agent can call a tool, a retrieval system decides which candidates to show it, and "most systems apply a fixed shortlist size to every query." Nine conditions across three tool-selection benchmarks (BFCL, MetaTool, ToolBench), three scorer types, and registries from 20 to 3,251 tools, scored with Bits-over-Random — a chance-corrected metric asking whether success at a given depth beats random selection at that same depth — and then the same signal reused as an RL reward for choosing depth per query.

The numbers

BFCL, 370 tools pooled into one registry so that every other function becomes a natural hard distractor (calc_area_triangle against calculate_triangle_area against calculate_area). The learned policy reaches 90.3 ± 2.4% at K = 7.4 ± 2.5 against 90.8% for a fixed shortlist of 50 — the same coverage from seven tools instead of fifty.

ToolBench, 3,251 tools. A fixed shortlist of five wins on aggregate coverage (64.7% against 61.9%) and finds nothing at all on the hard queries where the correct tool ranks 6th–20th; the adaptive agent finds 16.7% of those by searching deeper. Aggregate accuracy and hard-case accuracy point in opposite directions, which is worth carrying into any single-number claim about tool counts.

The downstream check is the part this wiki wanted. Validated with Claude Sonnet 4.6, shorter adaptive lists improved the model's own selection: 93.1% against 87.1% when always shown five, and 76.8% against 60.9% on medium-difficulty queries where the correct tool is present but not ranked first. Fewer, better-chosen candidates beat a fixed five — so the failure is not only retrieval missing the tool, it is the model choosing badly among what it was handed.

What it does to the standing question

attention-is-all-you-have crawled 56,804 published skills and argued they compete for fewer than a hundred reliable auto-trigger slots, a bound taken from the long-context literature rather than measured, with the authors proposing the experiment and not running it.

This paper does not run that experiment either — and it reframes it usefully. The variable that moves accuracy here is how many candidates are shown, not how many exist. A 3,251-tool registry is workable at K = 7 and unusable flat. So "how many skills can I install" is the wrong question wherever a retrieval layer stands between the registry and the prompt, and the right one is what that layer shows.

Which is exactly why the skills case is still open. Installed agent skills have no retrieval layer in the sense measured here — their trigger descriptions sit in the prompt together, so the installed count is the shown count. This paper measures the regime with a shortlist; the skills regime is the K = N corner it never visits.

The related work the authors summarise points the same way and is recorded as their citation, not as this wiki's reading: RAG-MCP "demonstrates clear degradation as N grows from 1 to 11,100 MCP tools", and LongFuncEval and Rabinovich & Anaby-Tavor find function-calling accuracy degrades as catalogues grow or as semantically similar tools are added.

Limits

The RL agent is "deliberately simple, serving as a probe of the metric rather than a proposed system" — the authors are not claiming a production method. Retrieval on BFCL is BM25 over function descriptions and parameter names, so a stronger scorer might move every number. Benchmarks are synthetic registries, not one team's installed toolset. And a single downstream model (Sonnet 4.6) supplies the selection-accuracy figures, so the 93.1/87.1 split is one model's behaviour, not a law.

Related

attention-is-all-you-have · model-context-protocol · agent-skills · synthesis