autoharness — a self-maintaining skill layer for Claude Code
A tigerless-labs plugin that treats an agent’s skill library as something to grow and prune from real usage, not hand-curate. It watches actual Claude Code sessions, distills the recurring ones into skills, folds new evidence back into skills it already wrote, and retires the ones that stop earning their keep — all without a daemon or a held-out benchmark. It is the self-improvement thread aimed at the whole library rather than a single skill, and the most concrete take yet on the spoke’s standing worry that self-authored capability drifts.
What it does
The pipeline runs over session data — user input, agent output, and tool interactions autoharness — and turns it into a managed skill set:
- Capture → distil. Session transcripts become candidate SKILL.md skills, so the source material is the agent’s own runs, not docs you point it at.
- Consolidate, don’t accumulate. Similar scenarios are grouped into one skill instead of piling up near-duplicates — the explicit answer to the bloat that an append-only skill library tends toward. New evidence updates an existing skill rather than spawning a variant.
- Lifecycle by invocation rate. It tracks how often each skill actually gets invoked and archives the under-used ones (archived, not deleted), so the library tracks current work.
- Ledger. An append-only ledger records each skill’s creation and every later edit, kept as metadata separate from the skill body so it doesn’t clutter what the agent loads at recall time.
The safety story
The drift/quality risk this spoke keeps flagging is handled by scoping and validation, not a review vote:
- It only modifies skills it authored itself — user-written and third-party-installed skills are off-limits. That bounds the blast radius of bad self-edits to the agent’s own output.
- Generated skills are validated before deployment.
- It runs daemon-free and benchmark-free: relevance comes from real invocation rates and real-time adherence monitoring of live sessions, not an offline test score.
Python 3.11+, zero third-party dependencies, wired into Claude Code’s plugin architecture and MCP autoharness.
Why it matters
The spoke has several instances of agents authoring skills — [[hermes-learn-skill-capture|Hermes
/learn]] converts a doc or a past conversation into one SKILL.md; adk self-generates
meta-skills; gstack‘s Reflect step codifies learnings. autoharness moves up a level: it manages
the library — discovery, consolidation against bloat, and usage-driven retirement — which
are the operations a growing self-authored skill set actually needs once authoring is solved. The name
is the tell: it auto-maintains the harness’s skill layer.
It also lands a fourth design point on the drift question. Where zouroboros gates self-edits through a three-model consensus vote, autonovel through an engineered LLM-judge, and autoresearch through an objective metric, autoharness uses the real invocation/adherence signal — does the skill get used, does the agent follow it — as its feedback, paired with author-only scoping + pre-deploy validation as the containment. That is the loop applied to a skill library, with usage as the truth signal.
Caveats
A project README (T3, self-reported, not yet inspected end-to-end or independently tested). Its headline claim — agent performance 42% → 78% “on benchmarks” through the maintained skill layer — is exactly the kind of unverified vendor metric the spoke’s Open questions / Benchmarks thread is waiting to see measured: no benchmark named, no neutral comparison, and the project itself disavows held-out benchmarks as its operating signal, so the figure sits oddly against its own daemon-free, benchmark-free design. Whether usage-rate consolidation keeps the library net-positive over time is the open self-improvement question, still unmeasured here.
Related
self-improving-agents · hermes-learn-skill-capture · agentskills-spec · agent-skills · agent-memory · zouroboros · autoresearch · loop-engineering · agentic-coding-harness · anthropic-skills · tigerless-labs