JetBrains — Ponytail skill for Claude Code: does it really cut tokens?

The independent measurement this spoke has been asking for since it opened. JetBrains ran a paired A/B benchmark of ponytail against stock Claude Code and published both the method and the numbers. T2: a third-party lab with no stake in the skill, testing someone else’s tool — the tier the standing Benchmarks open question was written to want.

Method

80 paired tasks from SkillsBench, auto-graded 0–1 with partial credit. Model: claude-sonnet-5 at medium reasoning effort. Harness: Harbor 0.18 with Docker sandboxes and task verifiers. Arm A was stock Claude Code 2.1.201; Arm B injected the Ponytail v4.8.4 ruleset via a SessionStart hook. 251 billed agent trials, USD 246.09, escalated through a 10-task smoke run and a k=3 validation stage.

Every trial was audited to confirm the ruleset reached the treatment arm (100%) and stayed out of the baseline (0%). That is precisely the contamination i-have-adhd‘s eval runner warns about in prose — an always-on flag leaking the skill into the baseline and making the comparison measure the skill against itself. Here someone checked, per trial.

Results

Against ponytail‘s advertised −54% code, −22% tokens, −20% cost, −27% time, the measured effect is roughly a third of the claim. JetBrains explains the gap without calling it dishonest: “Their −54% is a mean across twelve hand-picked feature tickets; ours is a median across 80 tasks nobody chose for this purpose.” Selection and statistic, not fabrication.

The verdict is a qualified yes — “Ponytail works… the first tool in this series that clearly saved money” — with the caveat “do not expect the advertised 54% everywhere.” Only the cost result clears conventional significance; the code reduction sits at p=0.088, and on the cost median a bootstrap interval “just touches zero.”

The effect is conditional

Savings concentrate where the baseline agent over-builds. On larger builds code fell 31%; on already-lean tasks the reduction was “near zero.” Ponytail doesn’t make an agent write less, it makes an agent stop writing too much — so its value is a function of how badly the baseline was over-engineering, not a constant discount.

That is a sharper claim than the skill makes about itself, and it generalises: an intervention that corrects a failure mode pays out in proportion to how often the failure occurs. Any restraint skill (output-shape skills as a class) should be expected to show this shape, and a single headline percentage will always be an artifact of which tasks were in the sample.

Two adherence failures

1. It doesn’t self-activate. Installed without the SessionStart hook, Ponytail fired zero times across ten sessions, despite a description claiming readiness for “ANY coding task.” The measured benefit therefore requires the always-on hook; the advertised install does nothing on its own. This is a concrete instance of the invocation problem — a skill’s description is a bid for the model’s attention, and this one lost.
2. The agent mostly ignored one rule. Ponytail asks for ponytail: comments marking deliberate shortcuts. Across 80 trials “that happened once.” An instruction in a loaded skill is not a constraint; compliance is partial and, until someone counts, invisible.

What it doesn’t establish

JetBrains states the limits plainly. SkillsBench measures task completion — not security, validation, error handling or accessibility, which is exactly the ground ponytail‘s guardrail promises to protect and this benchmark cannot check. 80 pairs can’t prove quality equivalence; the null is still compatible with a small degradation. And the post shows its own small-sample trap: the 10-task smoke run reported −3% code and a +9.6% cost increase, the opposite sign of the final result.

Related

ponytail · agent-evals · reasoning-scaffolds · i-have-adhd · cc-thinking-skills · jetbrains · agent-skills · synthesis