SKILLTRIAGE

An LLM tool that assigns a root cause to a confirmed skill-induced-failure, built by the authors of agent-skills-can-be-harmful to test whether their hand-built taxonomy survives being run as a procedure rather than used as a set of labels. Input is a target/reference pair already known to be skill-induced; output is a category, a subcategory, a natural-language reason, the cited skill section or trajectory step, and a repair suggestion.

How it works

Three stages. Input construction normalizes the pair into one task view plus two run views (loaded skill, verifier result, trajectory), with deterministic gates that reject a case lacking the minimum evidence. Differential evidence turns each taxonomy boundary into an explicit signal instead of asking the model to infer everything from raw traces — for functional failures, five checks covering environment change, whether that change was skill-prescribed or merely skill-induced, task path versus written path, whether the artifact was produced at all, and a focused construction-difference check that separates a wrong implementation from a missing one; for regressions, step-level token/time/tool records split into phases (exploration, pipeline, verification, dependency repair) and tagged by action, then differenced. Attribution hands the model the definitions, the normalized pair and the extracted evidence, and asks which label explains both the outcome and the divergence.

Runs on GPT-5.5, three times per case, aggregated 2-of-3.

Measured against the human labels

The two rows differ by 16 points because they have different amounts to go on. Functional attribution has a decisive artifact to point at — a wrong value, a missing field, a path that differs. Cost attribution has to decide which of several plausible expensive surfaces in one trajectory actually drove the bill, and a single run often contains dependency repair nested inside a verification loop, or build commands that could be counted as implementation or as checking. Errors in both rows land on taxonomy boundaries rather than scattering, which reads more like the categories touching than like the tool guessing.

What it does not do

It triages pairs that a human has already confirmed as skill-induced. Detection is still manual — something has to run the paired executions, spot the FAIL/PASS or the cost blowup, and discard verifier-narrow cases first. For the screening use the paper motivates it with (a marketplace checking new and updated skills continuously), the missing half is the expensive half.

Its own authors draw the design lesson rather than the accuracy one: an attribution tool should return the evidence — the skill section, the diverging step, the cost-heavy phase — not only the label, since the residual errors are exactly the cases where two labels fit the same evidence.

Related

agent-skills-can-be-harmful · skill-induced-failure · agent-failure-localization · agent-evals · agent-skills