Hunt Instead of Wait: Evaluating Deep Data Research on Large Language Models
Wei Liu, Peijie Yu, Michele Orini, Yali Du and Yulan He, arXiv 2602.02039 (submitted 2 February 2026, revised 15 May). Proposes Deep Data Research — autonomous insight extraction from databases — and DDR-Bench, a checklist-based benchmark for it.
The benchmark is not why it is filed here. Its Section 5.2 ablations are: they vary one harness mechanism at a time on a fixed model, which is what this spoke’s edge asked for and what scaffold-effects-gaia could not do.
The ablation design
One model throughout — Qwen3-Next-80B-A3B — under a deliberately minimal ReAct scaffold, with three separate ablations:
- reasoning budget (Table 3)
- memory mechanisms (Table 4)
- reactive versus proactive operation (Table 5)
The reported direction: memory and reasoning modifications produce inconsistent results rather than clear improvements. A separate appendix ablation compares whole framework additions, with numbers, below.
Why this is a different measurement from the one the spoke already holds
scaffold-effects-gaia crossed five models with three whole scaffolds (ReAct, Planner-Actor-Rater, planner-then-executor). That answers which architecture wins, and it cannot attribute the result to any component, because the three scaffolds differ in several ways at once. harness-bench has the same limitation at larger scale: 23.8 points between best and worst harness, no account of what produced the spread.
DDR-Bench inverts the design. One model, one scaffold, one knob moved at a time. That is the shape that can say memory did this, and it is the first such source here.
The paper’s own reason for a minimal scaffold, which cuts against this spoke
The design rationale is stated plainly:
“The system prompt follows a lightweight ReAct style and excludes explicit workflow, memory, or planning modules… DDR-Bench focuses on assessing model capabilities such as tool-use or long-horizon reasoning, without confounding effects from external scaffolding.”
Read that as a claim about evaluation and it is uncontroversial. Read it against this wiki’s subject and it is pointed: the authors treat harness machinery as confounding noise to be excluded when measuring a model. This spoke documents that machinery as the thing worth building. Both positions can hold — one is about isolating a model’s capability, the other about shipping a working system — but the corpus should notice that a benchmark designer’s instinct is to strip out exactly what harness builders add.
The framework ablation, with numbers
Added hours after this page was first written, once the PDF was extracted locally. The first version of this page said the paper contained no framework comparison. It does — Table A1, in the appendix, which the web extraction could not see.
Scores on three datasets × three models, ReAct as the baseline, each addition applied on its own (higher is better):
| Dataset | Model | ReAct | +Plan | +Memory | +Multi-Agent |
|---|---|---|---|---|---|
| MIMIC | Qwen3-4B | 16.67 | 8.14 | 11.46 | 4.44 |
| MIMIC | Qwen3-30B-A3B | 20.03 | 12.27 | 13.57 | 9.04 |
| MIMIC | GPT-5-mini | 28.81 | 23.67 | 22.22 | 12.66 |
| 10-K | Qwen3-4B | 40.94 | 14.25 | 17.43 | 26.50 |
| 10-K | Qwen3-30B-A3B | 42.33 | 47.59 | 37.10 | 31.80 |
| 10-K | GPT-5-mini | 46.35 | 49.82 | 45.35 | 30.04 |
| GLOBEM | Qwen3-4B | 26.21 | 22.76 | 23.45 | 22.30 |
| GLOBEM | Qwen3-30B-A3B | 35.63 | 25.75 | 22.30 | 23.91 |
| GLOBEM | GPT-5-mini | 36.09 | 28.05 | 28.72 | 25.64 |
Additions are Plan-and-Execute (planning), CoALA (memory) and AutoGen (multi-agent).
Two of 27 comparisons beat the plain ReAct baseline. Both are +Plan on 10-K, the financial-filing task. Memory never wins on any dataset with any model. Multi-agent is worst or near-worst in every row, and on MIMIC with Qwen3-4B it takes 16.67 down to 4.44 — a quarter of the baseline.
The paper’s own summary: “complex agents mostly degrade performance relative to the ReAct baseline, except for minor planning benefits in certain settings”, and “memory modules often destabilise performance.” Its explanation is that the extra machinery moves the model’s confidence, producing “premature self-termination or over-/under-thinking.”
What this does to the spoke’s thesis
This is the most direct evidence in the corpus that harness machinery can be negative-value, and it is not a small effect. harness-bench measured 23.8 points between best and worst harness and could not say what produced the spread; this says that on these tasks the spread is substantially damage, and that the plainest scaffold is the strong baseline.
Held with its limits. One benchmark, one task family (autonomous analysis over databases), three models, and the framework implementations are off-the-shelf rather than tuned. A tuned memory system built for the task is not what CoALA-out-of-the-box measures. But this spoke has spent months documenting mechanisms on the assumption that adding them helps, and the only controlled test of that assumption in the corpus says it usually does not.
Related
scaffold-effects-gaia · harness-bench · agentic-coding-harness · reasoning-scaffolds · synthesis