Agent evals

Evals are scored test runs of an agent: a dataset of inputs, a scoring function over the outputs, and a threshold. They exist because the usual check does not survive nondeterminism — an agent’s model may pick a different tool or a different action on identical input, so “a test that passes once offers no guarantee it will pass the next time” harness-agent-dlc. Pass/fail on one run measures luck. A graded score over a dataset measures the agent.

Eval score as a pipeline gate

The operational form is the one Harness productized in AI Evals: grade responses on correctness, safety and performance, then “wire that score straight into the pipeline as a pass-fail gate” trevor-stuart, so a regression blocks a release the way a failing unit test does. Two properties matter.

The gate must fire on model changes, not just agent changes — the same prompts and tools behave differently underneath a swapped or updated model, which is a dependency application code never had.

And it accepts a hard limit up front. The agent’s output stays irreproducible; what becomes reproducible is the record — every model call, every tool call, every step harness-agent-dlc. Determinism relocates from the artifact to the process and the trace. That is why eval gating arrives paired with agent tracing rather than replacing it.

The evidence problem

Evals are the discipline this wiki keeps finding missing — “no eval” sits among the nine failure modes in agent-anti-patterns, and eval-before-deploy is part of the loop-engineering discipline. But the corpus is equally full of reasons to distrust them:

cc-thinking-skills built a replication-gated eval harness over a 39-skill pack and found zero proven accuracy gains — an eval that returned a null result on its author’s own product.
cognition‘s stance is bluntly “trust no eval” (cognition-fable5-through-the-night), on the view that public benchmarks stop measuring anything once they are optimized against.
autonovel shows the other end: where no metric exists (prose quality), building a trustworthy evaluator becomes the research problem, answered with layered mechanical + LLM-judge checks.

So the honest position is narrow. An eval gate does not certify an agent is good; it catches a regression against your own dataset, which is a smaller and more defensible claim, and the one a delivery pipeline actually needs.

A score does not tell you what to fix

Even a trustworthy eval answers only did it regress. It does not say which component to repair, and the same visible failure can call for model post-training or a harness change depending on where it started model-or-harness-taxonomy. That is the gap agent-failure-localization fills — a post-hoc label naming the interaction the failure happened on and the component at fault. It presupposes the trace: you can only assign fault on an edge you recorded, which is the tracing argument on this page arriving from the diagnostic side rather than the reproducibility side.

Two altitudes of the same gate

The eval-as-delivery-gate here (harness-agent-dlc) is the outer-loop form. Its inner-loop twin is verification-loops-skills: the same “grade the work” instinct authored as a Claude Code skill that self-corrects during the session, before a PR exists. Its “on every PR” activation is where the two altitudes converge — a verification skill promoted into a pipeline gate.

Take the tally out of the model’s hands

The failure mode both forms share: a model asked to grade its own output can also hallucinate the grade. claude-security-plugin shows the fix as architecture — candidate findings survive a 2-of-3 verifier quorum, and “the tally is computed in Python by the report renderer, not asserted by the model.” Aggregation is programmatic and auditable, the same move as Harness scoring outside the agent (harness-agent-dlc) pushed down to each finding. If an eval’s verdict is itself model-asserted, it inherits the nondeterminism it exists to catch; a mechanical tally over independent judgments is the escape.

The gate applied to a skill, by its own author

i-have-adhd (2026-07-28) is the smallest instance in the corpus: a single skill shipping evals/ with cases, a blind weighted rubric and a release gate its own changes must clear — no blocking findings, correctness and safety each within 0.1 of baseline, weighted score above baseline. The author is gating his own skill against the version of the codebase without it.

The isolation discipline is this page’s leakage problem in miniature. The runner strips the operator’s own config (--setting-sources "", --ignore-user-config --ephemeral) because user-level plugins, hooks and memory otherwise contaminate every condition — the named worst case being the repo’s own always-on flag injecting the skill into the baseline, “making the comparison measure the skill against itself.” It also pins the model, since an unpinned eval silently re-runs against whatever the CLI defaults to next month. Both are the same failure this page keeps circling: an eval whose conditions aren’t controlled measures something, just not the thing claimed.

The gap: the harness exists, the results don’t. A gate nobody has published a run through is a proposal.

Someone else’s benchmark is the one that counts

jetbrains-ponytail-tested (2026-07-28) closes the loop this page keeps opening. JetBrains A/B-tested ponytail — a skill it doesn’t own — over 80 paired SkillsBench tasks and 251 billed trials, and audited every trial to confirm the ruleset reached the treatment arm and never the baseline. The contamination i-have-adhd‘s runner documents in prose, someone actually checked for.

The result is the argument for third-party evals in one line: −15.4% code (p=0.088) and −10.3% cost (p=0.004) against an advertised −54% and −20%. Roughly a third of the claim, and the discrepancy is methodological rather than dishonest — a mean over twelve chosen tickets versus a median over 80 nobody chose. Self-reported numbers aren’t usually fabricated; they’re measured on the sample that flatters them.

Two things worth carrying into any eval design here. The effect was conditional — 31% on over-built tasks, near zero on lean ones — so a single headline percentage is a property of the task mix, not the tool. And the run changed sign at small n: the 10-task smoke stage reported a +9.6% cost increase before the full run landed at −10.3%.

Name the residue you can’t grep

perplexity-api-platform-skills (2026-07-28) grades a code migration mechanically, then admits where that fails. check.sh runs leftover-pattern greps, required-pattern greps, and --live runs of the migrated app against production. Then a final rubric section headed “Judged by eye (not greppable)” names two things the script cannot catch: the related-questions feature must be “preserved via a workaround rather than silently dropped,” and the streaming consumer must exit “on every terminal event, not only response.completed.”

Both are precisely what a greppable rubric rewards. “No return_related_questions in the source” passes if the agent deletes the feature. So the authors mapped where their own gate could be satisfied vacuously and routed those cases to a human instead of pretending the script covered them.

Three more properties worth copying: the fixture is a production-verified app deliberately loaded with the hazards (and contributors are told not to tidy it); reference migrations are shipped but explicitly disqualified as answer keys — “Model output is nondeterministic, so treat these as examples, not golden diff targets” — which is this page’s nondeterminism problem given the right answer, grade properties not diffs; and the runbook documents the A/B against no-skill, telling you to re-run the task in a session without the plugin and grade both. A vendor publishing the procedure for measuring whether its own skill helps.

Still unpublished, like every eval here except jetbrains-ponytail-tested: the harness exists, no score does.

The deeper problem: a gate you can cheese isn’t a gate

Even a mechanical check has a floor: the agent can satisfy its letter while defeating its purpose (constraint-evading-behavior) — pass the type checker by weakening the type, pass the test by special-casing the input, suppress the warning instead of fixing the case. An eval’s worth is bounded by how hard it is to satisfy vacuously, which is why Cognition‘s “trust no eval” is rational and why the constructive fix is to design checks that shape the search space (invalid states unrepresentable) rather than merely reject at the end.

Two design traps, from a study that hit both (added 2026-08-03)

context-files-ablation-paper is the spoke’s most carefully powered evaluation, and its methodological findings outlast its result.

Screen tasks per agent, or measure nothing. A manipulation can only register on tasks whose outcome can move. Tasks at the floor or ceiling absorb it silently — which is why that study’s omnibus test reads p=1.00, a value its author calls “close to mechanical.” And the informative band is agent-specific: across 15 tasks both agents attempted, pass rates correlate at Spearman ρ=0.75 yet ~40% of tasks are borderline for one agent and floor/ceiling for the other. A borderline set calibrated on one harness gives another harness nothing to detect. See agent-specific-task-difficulty.

Task variance dominates, so scale tasks, not repeats. The paper’s Monte Carlo power analysis: at 17 tasks with 3 repeats, even a 30pp effect is caught 57% of the time; detecting 10pp at 80% power needs 120–200 tasks; going from 3 repeats to 10 lifts power for a 15pp effect only from 13% to 58%. Any eval reporting a small difference over a dozen-odd tasks is reporting noise, whichever direction it points.

Also from the same study, and cheap to inherit: turn counts are not portable across harnesses. Its effort classifier split on turns ≥30 and silently dropped eight high-effort Codex tasks, because Codex emits exactly one turn.completed event per session. Tool calls were the portable metric.

harness-agent-dlc · verification-loops-skills · claude-security-plugin · agent-guardrails · context-files-ablation-paper · agent-specific-task-difficulty · agent-loops-verification · loop-engineering · agent-anti-patterns · cc-thinking-skills · cognition-fable5-through-the-night · autonovel · i-have-adhd · ponytail · jetbrains-ponytail-tested · perplexity-api-platform-skills