Agent evals

Evals are scored test runs of an agent: a dataset of inputs, a scoring function over the outputs, and a threshold. They exist because the usual check does not survive nondeterminism — an agent's model may pick a different tool or a different action on identical input, so "a test that passes once offers no guarantee it will pass the next time" harness-agent-dlc. Pass/fail on one run measures luck. A graded score over a dataset measures the agent.

Eval score as a pipeline gate

The operational form is the one Harness productized in AI Evals: grade responses on correctness, safety and performance, then "wire that score straight into the pipeline as a pass-fail gate" trevor-stuart, so a regression blocks a release the way a failing unit test does. Two properties matter.

The gate must fire on model changes, not just agent changes — the same prompts and tools behave differently underneath a swapped or updated model, which is a dependency application code never had.

And it accepts a hard limit up front. The agent's output stays irreproducible; what becomes reproducible is the record — every model call, every tool call, every step harness-agent-dlc. Determinism relocates from the artifact to the process and the trace. That is why eval gating arrives paired with agent tracing rather than replacing it.

The evidence problem

Evals are the discipline this wiki keeps finding missing — "no eval" sits among the nine failure modes in agent-anti-patterns, and eval-before-deploy is part of the loop-engineering discipline. But the corpus is equally full of reasons to distrust them:

cc-thinking-skills built a replication-gated eval harness over a 39-skill pack and found zero proven accuracy gains — an eval that returned a null result on its author's own product.
cognition's stance is bluntly "trust no eval" (cognition-fable5-through-the-night), on the view that public benchmarks stop measuring anything once they are optimized against.
autonovel shows the other end: ~~where no metric exists (prose quality)~~ where the metrics that exist are not trusted (prose quality), building a trustworthy evaluator becomes the research problem, answered with layered mechanical + LLM-judge checks. Corrected 2026-10-11: prose-quality metrics do exist. EQ-Bench's Creative Writing v3 and Longform Writing boards score models with an LLM judge, and LitBench (EACL 2026) measured how far such judges agree with people: the best off-the-shelf judge matched human preference on 73% of story pairs, trained reward models on 78%. The gap autonovel works on is that roughly one verdict in four still disagrees with readers.

So the honest position is narrow. An eval gate does not certify an agent is good; it catches a regression against your own dataset, which is a smaller and more defensible claim, and the one a delivery pipeline actually needs.

A score does not tell you what to fix

Even a trustworthy eval answers only did it regress. It does not say which component to repair, and the same visible failure can call for model post-training or a harness change depending on where it started model-or-harness-taxonomy. That is the gap agent-failure-localization fills — a post-hoc label naming the interaction the failure happened on and the component at fault. It presupposes the trace: you can only assign fault on an edge you recorded, which is the tracing argument on this page arriving from the diagnostic side rather than the reproducibility side.

Two altitudes of the same gate

The eval-as-delivery-gate here (harness-agent-dlc) is the outer-loop form. Its inner-loop twin is verification-loops-skills: the same "grade the work" instinct authored as a Claude Code skill that self-corrects during the session, before a PR exists. Its "on every PR" activation is where the two altitudes converge — a verification skill promoted into a pipeline gate.

Take the tally out of the model's hands

The failure mode both forms share: a model asked to grade its own output can also hallucinate the grade. claude-security-plugin shows the fix as architecture — candidate findings survive a 2-of-3 verifier quorum, and "the tally is computed in Python by the report renderer, not asserted by the model." Aggregation is programmatic and auditable, the same move as Harness scoring outside the agent (harness-agent-dlc) pushed down to each finding. If an eval's verdict is itself model-asserted, it inherits the nondeterminism it exists to catch; a mechanical tally over independent judgments is the escape.

The gate applied to a skill, by its own author

i-have-adhd (2026-07-28) is the smallest instance in the corpus: a single skill shipping evals/ with cases, a blind weighted rubric and a release gate its own changes must clear — no blocking findings, correctness and safety each within 0.1 of baseline, weighted score above baseline. The author is gating his own skill against the version of the codebase without it.

The isolation discipline is this page's leakage problem in miniature. The runner strips the operator's own config (--setting-sources "", --ignore-user-config --ephemeral) because user-level plugins, hooks and memory otherwise contaminate every condition — the named worst case being the repo's own always-on flag injecting the skill into the baseline, "making the comparison measure the skill against itself." It also pins the model, since an unpinned eval silently re-runs against whatever the CLI defaults to next month. Both are the same failure this page keeps circling: an eval whose conditions aren't controlled measures something, just not the thing claimed.

~~The gap: the harness exists, the results don't. A gate nobody has published a run through is a proposal.~~ Corrected 2026-10-11: the results exist. The repo's evals/RESULTS.md, committed 2026-08-02, records the first run: 14 cases, 3 trials, claude-opus-4-8, the same model as blind judge. The skill raised the weighted score from 4.045 to 4.473 and cut blocking findings from 7 to 3, and the release gate failed anyway, because its "no blocking findings" rule is absolute. The author found that one case cannot be passed by any run (it asks the model to edit a repo while the runner disables tools), so as written no candidate could ever clear the gate. That is the useful result for this page: the first published run through a gate showed the gate was mis-specified. The author lists the limits too: three trials, and a judge from the same model family as the answers it grades. See i-have-adhd.

Someone else's benchmark is the one that counts

jetbrains-ponytail-tested (2026-07-28) closes the loop this page keeps opening. JetBrains A/B-tested ponytail — a skill it doesn't own — over 80 paired SkillsBench tasks and 251 billed trials, and audited every trial to confirm the ruleset reached the treatment arm and never the baseline. The contamination i-have-adhd's runner documents in prose, someone actually checked for.

The result is the argument for third-party evals in one line: −15.4% code (p=0.088) and −10.3% cost (p=0.004) against an advertised −54% and −20%. Roughly a third of the claim, and the discrepancy is methodological rather than dishonest — a mean over twelve chosen tickets versus a median over 80 nobody chose. Self-reported numbers aren't usually fabricated; they're measured on the sample that flatters them.

Two things worth carrying into any eval design here. The effect was conditional — 31% on over-built tasks, near zero on lean ones — so a single headline percentage is a property of the task mix, not the tool. And the run changed sign at small n: the 10-task smoke stage reported a +9.6% cost increase before the full run landed at −10.3%.

Name the residue you can't grep

perplexity-api-platform-skills (2026-07-28) grades a code migration mechanically, then admits where that fails. check.sh runs leftover-pattern greps, required-pattern greps, and --live runs of the migrated app against production. Then a final rubric section headed "Judged by eye (not greppable)" names two things the script cannot catch: the related-questions feature must be "preserved via a workaround rather than silently dropped," and the streaming consumer must exit "on every terminal event, not only response.completed."

Both are precisely what a greppable rubric rewards. "No return_related_questions in the source" passes if the agent deletes the feature. So the authors mapped where their own gate could be satisfied vacuously and routed those cases to a human instead of pretending the script covered them.

Three more properties worth copying: the fixture is a production-verified app deliberately loaded with the hazards (and contributors are told not to tidy it); reference migrations are shipped but explicitly disqualified as answer keys — "Model output is nondeterministic, so treat these as examples, not golden diff targets" — which is this page's nondeterminism problem given the right answer, grade properties not diffs; and the runbook documents the A/B against no-skill, telling you to re-run the task in a session without the plugin and grade both. A vendor publishing the procedure for measuring whether its own skill helps.

Still unpublished, like every eval here except jetbrains-ponytail-tested: the harness exists, no score does.

The deeper problem: a gate you can cheese isn't a gate

Even a mechanical check has a floor: the agent can satisfy its letter while defeating its purpose (constraint-evading-behavior) — pass the type checker by weakening the type, pass the test by special-casing the input, suppress the warning instead of fixing the case. An eval's worth is bounded by how hard it is to satisfy vacuously, which is why Cognition's "trust no eval" is rational and why the constructive fix is to design checks that shape the search space (invalid states unrepresentable) rather than merely reject at the end.

Two design traps, from a study that hit both (added 2026-08-03)

context-files-ablation-paper is the spoke's most carefully powered evaluation, and its methodological findings outlast its result.

Screen tasks per agent, or measure nothing. A manipulation can only register on tasks whose outcome can move. Tasks at the floor or ceiling absorb it silently — which is why that study's omnibus test reads p=1.00, a value its author calls "close to mechanical." And the informative band is agent-specific: across 15 tasks both agents attempted, pass rates correlate at Spearman ρ=0.75 yet ~40% of tasks are borderline for one agent and floor/ceiling for the other. A borderline set calibrated on one harness gives another harness nothing to detect. See agent-specific-task-difficulty.

Task variance dominates, so scale tasks, not repeats. The paper's Monte Carlo power analysis: at 17 tasks with 3 repeats, even a 30pp effect is caught 57% of the time; detecting 10pp at 80% power needs 120–200 tasks; going from 3 repeats to 10 lifts power for a 15pp effect only from 13% to 58%. Any eval reporting a small difference over a dozen-odd tasks is reporting noise, whichever direction it points.

Also from the same study, and cheap to inherit: turn counts are not portable across harnesses. Its effort classifier split on turns ≥30 and silently dropped eight high-effort Codex tasks, because Codex emits exactly one turn.completed event per session. Tool calls were the portable metric.

A pass rate hides both halves of the harm (added 2026-08-15)

agent-skills-can-be-harmful is the sharpest case yet for scoring being the wrong instrument. It audits skills by pairing every run against a no-skill or matched-skill reference, and the 307 confirmed harms it finds split into two things an eval cannot see:

Efficiency regressions pass. 182 of the 307 cases solved the task, at more than double the token or time cost. A pass-rate eval scores every one of them green. Cost has to be a graded axis, not a footnote, or the eval is blind to 59% of the damage in that corpus.
Functional failures average away. skillsbench reported gains from curated skills and a pass-rate drop on 16 of its 84 tasks. The mean can be up while a sixth of the suite goes backwards, and nothing in the score says which sixth or why.

The constructive half is the design: hold task, verifier, repository state, harness and model fixed, vary one configuration knob, and require the reference run to pass before charging anything to the knob. That converts "the score moved" into "this component did it" — agent-failure-localization by re-execution rather than by trace labelling. It is expensive (a second full run per audited case) and it needs deterministic verifiers, which is exactly why it ran on benchmarks and not on production traffic.

A product shipped the rig (2026-08-21)

agtx is the first tool in this spoke to ship a comparison harness for the methodology packs the spoke pages, rather than an eval for the agent itself. Its SWE-bench Lite runner takes a TOML config naming the workflow plugin, so superpowers, spec-kit, get-shit-done, OpenSpec, and skill stacks including ponytail become swappable arms over a fixed task, model and sandbox, each row reporting duration, tokens, dollar cost and a three-way outcome grade.

And every suite here scores one attempt. beyond-pass-at-1 tabulates it: swe-bench (20+ models), webarena and tau-bench all report a single try per task, none treats duration as a variable, and only tau-bench measures anything about repetition. Running an agent once and recording the result measures capability; a gate wants reliability, which is a property of the spread. That paper's 23,392 episodes at k=3 cost about $0.004 per episode, so the absence of variance from the standard suites is a choice about what to measure, not a budget constraint.

The choice of suite is itself a design decision the spoke should name. agtx picked SWE-bench Lite, which hands the agent a repository and a failing test. webarena (CMU, 2023) is the other shape: a self-hosted, reproducible environment — a real GitLab, a real forum, a real store — where the agent has to operate software rather than patch it, and the verifier checks world state instead of a diff. A harness that scores well on one says little about the other, and no source here runs both.

That cost column is the design point above made concrete: 🟡 (correct fix, no test written) against ✅ (fix plus test) separates outcomes a pass-rate eval collapses, and the published run has the ✅ arm costing 6× the cheapest 🟡. Whether that generalizes is unknown — the published table is one task per arm out of 300, one model, no repeats, run by the tool's own author, and every config disables the worktree parallelism agtx exists for. Read it as an instrument that now exists, not as a result.

The standing complaint this spoke inherited from llm-engineer-toolkit — the instruments exist in quantity and nobody publishes a result with them — needs a smaller amendment than it looks like. Somebody published. What is missing is anyone running it past n=1.

Single-run benchmarks measure the wrong property (2026-08-23)

science-of-agent-reliability states the charge against the way most benchmark pages in this spoke were built: a benchmark that reports one accuracy number from one run in a fixed environment "reveals nothing about whether an agent would succeed on the same task tomorrow, how it handles a slightly rephrased instruction, or how it adapts when its underlying infrastructure shifts." Its prescription is two-part — multi-run protocols that re-execute identical tasks for variance, and generative, parameterized benchmarks that rename fields, reorder responses, add API versions and inject fault probabilities, so the environment moves the way a deployed one does.

The twelve metrics are worth knowing as a menu even for an eval nobody publishes: consistency (outcome, trajectory-by-distribution, trajectory-by-sequence, resource), robustness (fault, environment, prompt), predictability (calibration, AUROC, Brier), safety (compliance, harm severity). The trick that makes them comparable across capability levels is normalizing variance by p(1−p) and expressing robustness as a ratio against the unperturbed run — a weak agent is not scored twice for being weak.

The paper also maps three real incidents onto the metric that would have caught them: Replit's deleted database to harm severity plus prompt robustness, OpenAI Operator's unconfirmed purchase to compliance plus trajectory consistency, the NYC chatbot's confident wrong legal guidance to calibration plus outcome consistency. Read that table as a design aid, not as evidence — the counterfactual is the authors' reading of failures they did not observe.

science-of-agent-reliability · agtx · webarena · beyond-pass-at-1 · harness-agent-dlc · verification-loops-skills · claude-security-plugin · agent-guardrails · · stax agent-skills-can-be-harmful · skill-induced-failure · skillsbench · swe-skills-bench · context-files-ablation-paper · agent-specific-task-difficulty · agent-loops-verification · loop-engineering · agent-anti-patterns · cc-thinking-skills · cognition-fable5-through-the-night · autonovel · i-have-adhd · ponytail · jetbrains-ponytail-tested · perplexity-api-platform-skills