Agent-specific task difficulty
The same coding task can be trivial for one agent and impossible for another, so which tasks are informative depends on which agent you are testing. From context-files-ablation-paper, where it is both a finding and a warning about how the spoke’s other evaluations were built.
The measurement
Across 15 tasks attempted by both Claude Code (sonnet-4-6) and Codex CLI (gpt-5.5), per-task pass rates correlate positively but imperfectly: Spearman ρ=0.75, p=0.001 (Pearson r=0.77). So difficulty largely transfers — but the borderline band does not. Of those 15 tasks, 6 are borderline (0 < pass < 1) for exactly one agent, and 6 differ in floor/ceiling status between agents. For roughly 40% of tasks, the agent that would reveal an effect is not the agent you ran.
Two concrete cases from the paper: pdm#3790 is borderline for Claude (1/3, 2/3, 1/3 across
strategies) and a flat floor for Codex (0/3 everywhere). pdm#3769 inverts it — Claude passes 3/3
under every strategy while Codex sits at 2/3, 3/3, 1/3.
Why it matters for any agent evaluation
A manipulation can only show an effect on tasks where the outcome can move. Tasks at the floor (never passes) or ceiling (always passes) absorb any intervention and report nothing. This is the structural reason context-files-ablation-paper‘s omnibus test reads p=1.00 — the author calls that value “close to mechanical,” since there is almost no marginal variance to permute.
The operational consequence he draws: ablation studies must screen tasks per agent. A borderline set calibrated on one agent hands another agent mostly floor/ceiling work. His own study inherited exactly this — the 4 screened borderline tasks were calibrated on Codex, so they are not necessarily borderline for Claude.
It also supplies a candidate explanation for why the field’s published results on context files disagree: if one study drew tasks from Codex’s informative band and another from Claude’s, they would reach opposite conclusions without either being wrong about its own setup. Offered as a hypothesis in the paper, since the prior studies’ exact task sets aren’t available.
A second portability trap from the same study
Related in kind, and worth carrying: the author’s effort classifier split on turns ≥ 30 and
silently marked every Codex task “trivial”, because Codex emits exactly one turn.completed event
per session. Eight genuinely high-effort tasks (18–51 tool calls, 410–1220s) were dropped before
anyone noticed. Reclassifying on tool calls recovered them.
Turn-based metrics are not portable across agent architectures. A harness that reports one turn per session and a harness that reports one per exchange are not measuring the same unit, and any threshold over that unit silently means something different in each.
How this wiki should read its own evidence
The spoke’s other measured sources — cc-thinking-skills, jetbrains-ponytail-tested — each run on a single harness. Nothing here says their results are wrong, but this page is the reason to stop reading a single-agent result as a statement about agent tooling generally. The unit that generalizes is the mechanism, not the number.
Related
context-files-ablation-paper · agent-evals · agents-md · cc-thinking-skills · jetbrains-ponytail-tested · agentic-coding-harness · synthesis