Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebases

Sherman Wong, Zhenting Qi, Zhaodong Wang, Nathan Hu, Samuel Lin, Jun Ge, Erwin Gao, Wenlin Chen, Yilun Du, Minlan Yu and Ying Zhang — Meta and Harvard, arXiv 2512.10398, v5 dated 20 December 2025. Read via the arXiv HTML.

The second harness ablation this wiki holds, and the one on a coding workload that ddr-bench left the corpus asking for.

The scaffold

Four named mechanisms, which is itself useful — most harness write-ups here describe an architecture without separating the parts that could be removed:

• F1 Context Management — hierarchical working memory with adaptive context compression driven by a planner agent.
• F2 Note-Taking Agent — persistent cross-session memory written as structured Markdown.
• F3 Extensions — modular tool handlers for file editing, CLI execution, code search and prompt shaping.
• F4 Meta-agent — an automated build-test-improve loop that synthesizes and refines the agent’s own configuration.

Headline result: 54.3% Resolve@1 on the full SWE-Bench-Pro with Claude 4.5 Opus, which the authors report as ahead of prior research baselines.

The ablation, which is what the edge asked for

Table 2, on a 100-example SWE-Bench-Pro subset, varying context management and tool sophistication on a fixed model:

Two numbers carry the finding. Context management is worth +6.6 points on Claude 4 Sonnet (42.0 → 48.6) and +0.6 on Claude 4.5 Sonnet (51.0 → 51.6). Tool sophistication is worth +7.0 on Claude 4.5 Sonnet (44.0 → 51.0), and the authors describe removing the learned tool-use features as causing a large decline independent of context management.

So the same mechanism is worth ten times more on the weaker model than on the stronger one. That is the shape of the result, and it is a harness-versus-model finding rather than a harness finding: a scaffold component that looks decisive on one model nearly vanishes on its successor. Anyone reading a harness’s reported gain has to ask which model it was measured on, and nothing in the vendor material this spoke holds reports that pairing.

Against DDR-Bench

ddr-bench varied reasoning budget, memory and reactive-versus-proactive one at a time on a fixed model (Qwen3-Next-80B-A3B) and reported inconsistent results, not clear improvements — but its tables would not extract, so this corpus held the design and no deltas.

Now there are deltas, from a different lab, a different workload (software engineering rather than deep data research) and different models. They do not contradict DDR-Bench so much as explain how its conclusion is reachable: if a mechanism’s value collapses as the base model improves, then an ablation run on one model will read as “inconsistent” from the outside. Two ablations, two labs, and the common thread is that per-mechanism gains are not stable across models.

Limits worth stating

The ablation is on a 100-example subset, not the full benchmark, and the paper reports no variance or repeated runs — single numbers on a small sample, where a 0.6-point difference is not distinguishable from noise. Only two of the four mechanisms (F1, F3) are varied; the note-taking agent and the meta-agent are never removed, so the corpus still has no measured value for persistent memory or self-configuration. And it is a preprint at v5 from the lab that built the system, evaluating its own scaffold.