Scaffold Effects on GAIA: A Controlled Comparison

Jason Starace, arXiv:2606.08529 (2026-06-07). The design this spoke has asked for since June and been handed seven near-misses of: five models crossed with three scaffolds on the same task set, pre-registered, with both factors varied in one experiment. It is the first source here that can say what a harness buys relative to a model upgrade rather than in isolation.

The design

GAIA validation Levels 1 and 2 — 53 + 86 = 139 questions. Five models across three providers (Claude Opus 4.7, Sonnet 4.6, Haiku 4.5; Gemini 3.1 Pro Preview; GPT-5.5) × three scaffolds × 139 questions × 3 attempts = 6,255 designed runs, 5,907 completed. The scaffolds:

• S1 · ReAct — the plain interleaved reason/act loop.
• S2 · Planner-Actor-Rater — a multi-agent split with a critic.
• S3 · Planner-then-executor — plan once, then execute.

The hypotheses were pre-registered, including the headline one (that scaffold choice alone moves at least one model by ≥10 points on Level 1). That matters for a claim this spoke has only ever seen asserted after the fact.

The result

Accuracy, primary slice:

Three things fall out, and they do not all point the same way.

Scaffold choice is worth a lot within one model. Haiku swings 23 points on L1 (0.459 → 0.226) and 20 on L2 purely by scaffold. The paper’s abstract reports a gap of up to 28 points within a single model; the primary-slice table above tops out lower (Opus L2 spans 14), so the 28 comes from a slice other than this one — worth noting rather than repeating as the headline.

But the scaffold does not substitute across a capability tier. Haiku’s best configuration (0.459 L1) never reaches Sonnet’s worst (0.660). No amount of structure lifts the small model into the next model’s range. This is the sharpest bound the corpus has on “structure substitutes for capability”.

It does substitute across one step — which is exactly agentsys‘s claim. Sonnet + PAR (0.730 L1, 0.620 L2) beats Opus + planner-executor (0.704 L1, 0.655 L2) on Level 1 and beats Opus + ReAct (0.558) on Level 2. The “cheaper model plus a better harness beats the expensive model” claim this spoke has carried as an unverified README line since June gets its first controlled support — and its first boundary: it holds one tier down, not two.

The best scaffold is model-specific, and sometimes it is the simplest one. Opus is strongest on L1 with plain ReAct, and the multi-agent scaffold hurts it there. The multi-agent advantage showed up only within the Anthropic family and did not transport across providers. Cheapest and most accurate at L2 was Gemini + planner-executor at $0.14–0.23 per correct answer, against $1.20 for the worst cell.

What it costs the reader to believe

The limitations are stated and two of them bite:

• Tool-surface confound. S1 used a different tool set from S2/S3, so loop structure and tool surface vary together. The scaffold effect is really a scaffold-plus-tools effect.
• Validation contamination. GAIA validation answers are public; the author says high L1 accuracy “may partially reflect memorization rather than capability.”
• Provider-side failures hit 314 records (Anthropic deserialization), credit exhaustion created asymmetric retry budgets across providers, and cost is log-derived rather than billed.

The conclusion the author draws is the one this spoke should adopt: a single-scaffold benchmark number is a scaffold-conditional estimate, not a capability measurement — and the elicitation gap is not guaranteed to close as models improve.

Standing

T1 — pre-registered, crossed design, ~5,900 runs, limitations published. Single author, single benchmark, and GAIA is a general-assistant task set rather than a coding one, so it does not settle the coding-agent case that most of this spoke’s harnesses target. It closes the design gap; the domain gap stays open.