VibeSearchBench

VibeSearchBench: Benchmarking Long-horizon Proactive Search in the Wild — Xiaohongshu Dots Studio and Unipat AI, arXiv:2605.27882. Read in full. The first source this wiki holds that scores what a search agent gathered rather than what it wrote, which is what growth edge 2 has been asking for since 2026-08-07. It does not close the edge, for reasons below, but it breaks the pattern the three previously hunted benchmarks all shared.

Why it is different

The authors start from an "evaluation–experience gap". Frontier models climb BrowseComp and WideSearch while users say the results are off-topic. Their diagnosis is that the benchmarks give the agent a fully-specified query up front, allow one turn, and score against a fixed schema. So VibeSearchBench does none of those:

• 200 manually curated bilingual tasks across 20 domains, split into a professional and a daily-life subset, each written by a domain annotator who designs a vague opening query, plays out a realistic multi-turn refinement, and then builds the answer graph.
• Each task carries a user persona and a progressive-disclosure simulator that only reveals the next need when the agent triggers it, so intent has to be elicited rather than read off the prompt.
• The output is a schema-free knowledge graph of triples, scored by graph matching against an expert-built ground-truth graph.

That last part is the one that matters here. Recall asks whether each ground-truth triple is covered by the predicted graph — allowing direct matches, subsumption, coverage by several triples together, or derivation through predicted relations — and precision is the fraction of predicted triples that participate in covering at least one ground-truth triple. An LLM judge does the entailment so aliases and synonyms don't fail on string equality. This is a measurement of the evidence an agent came back with, per item, against a known target set.

The numbers

Seven frontier models, run under both ReAct and the OpenClaw harness. Best average F1 = 30.30 (Claude Opus 4.6, OpenClaw); every model below 33. The shape of the errors is more useful than the ranking:

• Recall exceeds precision almost everywhere. Claude: P 24.88 / R 36.34, and on the daily subset P 21.60 / R 39.20 — broad sweeps that drag in irrelevant material. The exception is Gemini-3.1 Pro (P 34.61 / R 20.63), which is conservative and leaves nearly 84% of the professional subset's ground-truth triples unrecovered.

• Proactiveness tracks score. The highest-scoring model also runs the most independent work per user turn (7–8 tool calls, ratio 8.26); the lowest ratio (2.84) waits to be driven and gets severely limited coverage.

• Trajectory compression costs 8–12 F1 points, and no model ever reached the simulator's completion signal.

• Ablating three OpenClaw mechanisms — sub-agent collaboration, local memory and life-long memory — yields no significant improvement on any of them.

What it means for this wiki

Read against llm-osint-reliability-study and llm-geolocation-test, which both score analysis of supplied evidence, this is the missing stage measured at last: a collector's output checked item-by-item against a set somebody defined in advance. And the result is that the best frontier model recovers roughly a third of what an expert said the task required, while producing three irrelevant triples for every one that lands.

For llm-osint and kallisto-osinter that is a sobering prior, and it is a prior rather than a finding, because two things separate this benchmark from OSINT work:

1. The domain is consumer search — travel, fashion, healthcare, gaming, not adversarial collection against a subject who may be actively obscure. Nothing here is hidden on purpose.
2. Collection is not autonomous. A simulated user sits in the loop disclosing needs, so what is measured is a collaborative session, not an agent left to run. That is the residue growth edge 2a now carries.

The authors' own conclusion points at model capability rather than tooling: harness features made no difference, so the ceiling is in long-context reasoning and intent elicitation. That is a direct second data point for ../agentic-tooling-wiki's finding that scaffold memory does not buy what it promises — arrived at independently, on a different task, with a different metric.

Provenance note. The authors are a search-and-social company and their benchmark is not peer reviewed. They ship none of the seven models scored, and the ranking puts three competitors' models above two others' with no product of their own in the table, which is why this is graded T1 as a dataset/benchmark primary source. The grade would not survive if a Xiaohongshu agent appeared in the results.

Paired 2026-10-01. llm-deanonymization-at-scale measures the other half of the setup this benchmark leaves out: an agent with no user in the loop, against subjects whose footprint was thinned on purpose. It scores one fact per subject where this benchmark scores a whole graph.