The Hidden Footprint: Making Storage a First-Class Metric for LLM Agent Evaluation

Chenglin Yu, Hongquan Gui, Ying Yu, Tao Zeng and Ming Li — Hong Kong Polytechnic University (Industrial and Systems Engineering; Research Institute for Generative AI), Zhejiang Normal University, and Shanghai Juepei Flexible Supply Chain Technology. arXiv 2607.11149, v1 13 July 2026, v3 23 July 2026. T1 — read from the PDF, extracted locally. None of the five authors ships any of the eight frameworks measured, which for a cross-framework benchmark is the fact that decides the grade.

This is the first source in the corpus that measures the framework layer. agent-framework has been able to say the layer is crowded, Python-shaped and specializing, and unable to say whether any of it helps — both censuses it rests on grade on ergonomics and community size. AgentFootprint does not answer whether a framework helps you succeed. It answers a question nobody had asked: what does each one leave on disk, holding the work identical.

The control that makes it a comparison

The measurement problem with any framework comparison is that two runs differ in what the agent did as well as in what the framework did. The paper separates them with a fixed-trace control — one scripted trajectory (three reads, 182.7 KB of observations, one answer) replayed through each framework's own persistence adapter, same model, same tools, same task.

Identical work in, and retained bytes come out differing by 6.7×.

The stronger version drops the scripting. Across configurations that all reach 100% accuracy on the same file-QA suite, retention differs by 15.7× — LangGraph 5.10 MB against CrewAI 0.32 MB. Same answers, same score, sixteen times the residue.

Eight frameworks, seven of which persist anything

Versions are pinned, and the paper is careful that this is the point: results describe versioned configurations, not frameworks in the abstract.

The abstract's "seven persisting frameworks" and the table's eight rows are the same fact: smolagents retains nothing, which is why it has no duplication factor and no growth exponent. That is a design position, not an oversight, and the paper refuses to score it as a win — retaining nothing means recovering and resuming nothing, which is what the reconstructability column is for. CrewAI shows that cheapness is not one thing: 0.32 MB but R=1, while OpenAI Agents SDK sits at a comparable 0.33 MB with full R=3. Nearly the same bytes, and one of them can rebuild the history.

Read the two columns together or not at all. The authors state the gaming argument themselves: retention and reconstructability can each be maximized trivially in opposite directions, and the target is exact reconstruction at minimal bytes. A leaderboard on either alone would be worse than none.

Growth is superlinear, which is the part that bites later

The growth exponent α is fitted from retention against round count at T ∈ {25, 50, 100, 200}. Frameworks that persist the full history at every step run up to α = 1.95 — near-quadratic — and the worked consequence is the paper's most quotable number: a 200-round monitoring loop over a single 2 KB status file leaves 323 MB. Windowed context management stays near-linear.

On a deliberately minimal write task, framework residue exceeds the delivered output files by up to 24,033×. The artifact the agent was asked to produce is a rounding error against the trace of producing it.

The measurement trap, which is a finding in itself

Naïve byte-level measurement understates duplication by an order of magnitude, because database paging and JSON escaping break up repeated content so a byte-level differ cannot see it. The suite is serialization-aware for that reason: it chunks logical streams (SQLite cells, JSONL records) and hashes them with SHA-256 before comparing.

This generalizes past the storage question. A metric that under-reports because its subject is stored in a form the instrument cannot parse is the same failure this hub keeps finding in its own gates, most recently on 2026-08-21 in check:tiers. The defect is never that a number is wrong; it is that nothing indicates the number is wrong.

The public evidence agrees, from outside the lab

The controlled suite is synthetic, so the paper goes and looks at real submissions: 108 instance-normalized SWE-bench Verified submissions (swe-bench), whose exported trajectories span 1,617× — three orders of magnitude — in per-instance volume, with no detectable correlation with resolve rate.

That is the finding that generalizes. Whatever storage buys, it is not measured by success on the benchmark this spoke's evaluation thread is built on. And it means the dispersion is not an artifact of the authors' harness: it is visible in work people published for other reasons.

A production deployment audit adds the composition. 442 MB across eight tasks, of which raw traces, service logs and debug logs are 290.6 MB (66%) and conversation stores are 1%. Per-task retention has a heavy tail — median 715 KB, P90 81.4 MB.

Most of it is removable

A content-addressed store cuts retention 4.8–32.7× while every restoration test passes and every reconstructability score is reproduced. So the dispersion is not a recovery/economy trade-off being priced differently by eight reasonable teams. It is mostly duplication that nobody had a reason to look at, and exact history reconstruction does not require megabytes.

The reason to care is stated as retention obligation rather than disk cost: at 10⁴ tasks/day, measured defaults produce 51 GB/day against 3.1 GB/day for the history-equivalent form at equal R — a 16× gap in what a compliance regime would oblige you to hold, delete and produce.

Does the model underneath matter? Barely

Full-roster replication preserves the storage ranking across backends: ρ = 0.96 on Qwen3.6-27B (sizes 0.93–1.06×), and ρ = 0.89 on a weak Qwen2.5-7B despite its accuracy collapsing. Persistence design dominates backend choice. Containerized and sandbox measurements differ by 0–8.5%.

Scale of the study: a 509-run main study plus 552 replication and variant runs, 1,061 sandboxed runs in all.

Limitations, audited by the authors

Six boundaries, stated: (i) the controlled suites cover retrieval and retention, not general reasoning; (ii) results describe versioned configurations, not frameworks in the abstract; (iii) the wild data are exports, not runtime residue; (iv) R covers conversation history only; (v) the roster is representative and α is workload-conditioned; (vi) fresh sandboxes understate shared threads (95.2 vs 5.8 MB). AutoGen's save cadence is flagged by the authors as their most contestable configuration choice; an end-of-run variant lands similarly (2.1 vs 2.49 MB) and stays superlinear (α = 1.50).

The second boundary is the one this spoke should hold onto. Version-pinned results decay, and agent-framework's point that this category has no agreed membership applies here too — eight is another census, chosen by another author, and smolagents and Agno being on it while DSPy and the vendor SDKs are not reflects a vantage point like all the others.

What it does and does not close

It closes the complaint that nobody measures this layer: here is an outcome, held against constant accuracy, with a control, a replication and public corroboration. It does not close the question agent-framework actually asks — whether a framework helps at all against building directly on an API — because accuracy here is a control variable, deliberately pinned at 100% so storage can vary. The comparison the spoke wants would let correctness move.

So the layer has now been measured exactly once, on an axis nobody proposed until this paper, and the axis turns out to be uncorrelated with the one everybody else reports. Whether storage is the only thing about frameworks that can be measured, or the first of several, is open.

Related

agent-framework · beyond-pass-at-1 · confucius-code-agent · swe-bench · durable-agents · agent-orchestration · agentic-coding-harness · synthesis