Do Context Files Help Coding Agents? (Khatri, arXiv 2607.27250, 2026-07-28)

The first controlled experiment this wiki holds on whether AGENTS.md / CLAUDE.md files actually improve coding-agent output. Prakhar Khatri, Independent Researcher, cs.SE + cs.AI, submitted 2026-07-28, code and data released.

The finding: context-injection strategy does not measurably move correctness on either agent. Not “helps a little.” No detectable effect, bounded to ≤10pp for Claude Code and ≤15pp for Codex.

The design

Three injection strategies, run over the same tasks:

• NONE — the AGENTS.md is deleted from the workspace; the agent works from the codebase alone.
• ALWAYS ON — the full file is injected into the system prompt every turn, and removed from the workspace so it cannot be read twice.
• SELECTIVE — topic-organized wiki/*.md files sit in the workspace with a system-prompt hint to consult them; the agent retrieves on demand with its Read tool.

Two agents from different providers: Claude Code (claude-sonnet-4-6, capped $6/run) and Codex CLI (gpt-5.5, capped at 120 turns). 17 tasks from 3 Python repositories — pdm (477-word context file), firebase-admin-python (1,236 words, rated “Excellent” on the author’s rubric) and opshin (248 words) — 15 shared plus 2 Codex-only. 3 repeats per cell, 291 runs completed, 288 evaluated.

Tasks are mined from merged pull requests: the PR description is the prompt, the base commit the starting state, and the PR’s own test files the hidden oracle. SWE-bench Tier-C protocol — the agent never sees the tests, and passes only if all gold tests pass. Runs execute on an egress-locked pod with GitHub DNS blackholed, credentials stripped, push/commit denied via PATH shims, and future commit history pruned so the agent cannot read the gold solution out of git log.

The numbers

Every Claude pairwise difference is ≤2.3pp. Codex’s largest is 5.9pp and points the wrong way — NONE beats SELECTIVE. On the 4 Codex-borderline tasks chosen specifically to have room to move (17–67% baseline pass rate), NONE scores 58% against 42% for both context arms.

The author does not oversell the statistics. He states that the omnibus p=1.00 is “close to mechanical” because the floor/ceiling structure leaves almost no marginal variance to permute, and that the TOST bounds are “descriptive equivalence… not a powered equivalence claim.” The power analysis is blunt about it: at n=17 with 3 repeats, even a 30pp effect is caught only 57% of the time, and detecting 10pp at 80% power would need 120–200 tasks. Adding repeats barely helps — task-level variance dominates, so scaling means more tasks, not more runs.

Why the null holds: skill, not knowledge

The mechanism is the part that generalizes past this sample. The author triaged the near-miss failures — the ones most likely to flip on one extra fact:

• opshin 510 — built the entire union-expansion optimization pass, introduced a correctness bug. Gap: engineering precision.
• firebase 907 — needed proactive auth-token refresh, implemented reactive retry. Gap: architectural pattern choice.
• opshin 554 — knew the V3 rule from the code, miswired the check. Gap: exact behavioral specification.
• opshin 593 — needed type-narrowing through isinstance + assert. Gap: type-system reasoning.

None of them is a knowable-fact gap an AGENTS.md could fill. They fail on implementation skill — feature design, pattern selection, exact wiring — not on missing repository-private knowledge.

The manipulation probe

The obvious objection to any null is that the manipulation was inert. The author pre-registered a probe against exactly that: re-run the two convention-closest near-misses under all three strategies, 3 repeats, on both agents — 36 cells.

The real, unmodified AGENTS.md never converts a near-miss to a pass on either agent. Codex fails 18/18 regardless of strategy; firebase 907’s NONE near-miss (109 tests pass, 1 fail) never crosses. On the single task with cross-agent dynamic range, Claude passes 907 at 2/3 under NONE, 1/3 under ALWAYS ON, 0/3 under SELECTIVE — the author explicitly declines to claim a general downward effect from one task at n=3, while noting the trend is “clearly non-positive.”

His conclusion is precise: the manipulation is not inert (it can perturb behavior, and the files rate Good/Excellent on his rubric) — “it simply does not supply the implementation skill that gates these tasks.”

What context did change: process, not outcome

Two narrow effects survive, both about how the agent works rather than whether it succeeds.

Cache footprint. Claude’s SELECTIVE uses significantly less cache-creation tokens than NONE (11/11 tasks lower, p=0.001, Holm-corrected 0.012). The author reads this mechanically: SELECTIVE keeps a short hint in the prompt while ALWAYS ON re-presents the whole file every turn, so the difference follows from delivery mechanics, not from the agent being more capable.

Blind full-suite test runs. On opshin — the one repository whose file warns that “the full test suite takes >20 minutes” — Claude’s blind full-suite pytest invocations fall monotonically: NONE 3.67 → ALWAYS ON 2.44 → SELECTIVE 1.67 per cell, with wall-clock 2689s → 2066s → 2032s. Strip the warning and the agent repeatedly runs the slow suite; give it the warning and it runs targeted tests. Labelled exploratory — post-hoc, n=4–5, and absent on firebase, which shows the opposite direction.

So a context file bought a real operational saving here, and it came from one specific warning about a cost the agent could not observe, not from the conventions or architecture guidance.

Reconciling the prior contradiction

The paper exists because two 2026 studies disagreed: Lulla et al. found AGENTS.md improves agent efficiency, Gloaguen et al. found no effect on completion. Khatri’s proposed reconciliation, offered as a hypothesis since he lacks their task sets: correctness genuinely doesn’t move (Gloaguen), the efficiency gain is the mechanical cost of injecting more tokens rather than a capability change (and Gloaguen reports the opposite sign on cost — context files raising inference cost by over 20%), and the residual disagreement comes from agent-specific-task-difficulty.

What the author tells practitioners

Scoped to his setting — 3 Python repositories, naturalistic style-guide context, varying the delivery channel rather than the content — “generic context files do not measurably improve coding-agent correctness.” The safe reading he offers: effort spent on generic context documents may pay off less than effort on task decomposition, tooling, or example-driven prompting. He explicitly does not rule out benefits for other languages, larger repositories, or purpose-built, task-specific context, and names that as the open question.

Limitations, in his own accounting

Eight are listed. The ones that bite hardest here: n=15–17 tasks with MDE >30pp; three Python repositories only; injection-channel asymmetry (Claude gets a system prompt, Codex a user-turn prepend, because Codex has no system-prompt flag) — a confound between agents, though the within-agent comparison stays clean; and ecological validity, since ALWAYS ON injects harder than the natural workflow where a file just sits in the workspace. He argues that if guaranteed presence doesn’t help, natural discovery can’t either, and marks that as an inference rather than a measurement — a fourth “natural” arm would settle it.

The SELECTIVE arm carries a confound he flags twice: its wiki is content-matched to the AGENTS.md for opshin only. For pdm and firebase it is a broader auto-generated repository wiki, roughly 10× and 18× the words. So SELECTIVE varies both channel and corpus — which, as he notes, only strengthens the correctness null (strictly more material, no more passes) while making the cache result unattributable to channel alone.

Also: claude-sonnet-4-6 and gpt-5.5 as of this study, so the null is a snapshot.

The untested case has a specimen (added 2026-08-03)

The paper’s own carve-out is purpose-built, task-specific context — “a fact the agent provably cannot infer” — which it names as future work. claudecode-writer is a clean instance of that shape: its context/writing-examples.md holds two or three samples of the user’s actual posts, the README calls adding them the most important setup step, and its answer to “Claude doesn’t sound like me” is add more examples.

Two things separate it from what was ablated here. It is context as few-shot demonstration rather than as instruction, and voice is a genuine knowledge gap — precisely the category the failure triage found absent from real coding tasks, where near-misses turned on implementation skill instead. So the mechanism argued for the null does not obviously apply, and this is the comparison a follow-up study wants.

Also unmeasured: claudecode-writer publishes no evaluation, and style match has no gold test to run against. A well-posed instance of the open question, not an answer.

Tier

T1 — primary research artifact: full method, pre-registered probe, released code, data and power-analysis. The tier rates the artifact, not the strength of the conclusion. It is a preprint (not peer-reviewed), single-authored by an independent researcher, and by its own power analysis cannot detect anything under 30pp. What it establishes is a bounded null with a mechanism, which is a different and more useful thing than a bare negative result.

Related

agents-md · agents-md-spec · context-driven-development · agent-evals · agent-specific-task-difficulty · cc-thinking-skills · jetbrains-ponytail-tested · agentic-coding-harness · claude-code-best-practices · synthesis