cc-thinking-skills

Claude Code Thinking Skills (TJ Boudreaux, MIT, 1,471 stars / 172 forks / 4 open issues as of 2026-10-04; v1.0.0 released 2026-08-04, last push 2026-08-07) is a catalog of 28 mental-model and critical-thinking frameworks (cut from 39 in the August catalog cutover) shipped as agent skills — first-principles, inversion, pre-mortem, Bayesian updating, Cynefin, theory of constraints, OODA, TRIZ, Fermi estimation, red-teaming — plus a thinking-model-router meta-skill that reads a problem and returns NONE, one skill, or up to three. Installable through npx skills add, the Claude Code plugin marketplace, or by copying skills/* into ~/.claude/skills/; the README now targets Copilot, Codex and Cursor as well.

The skills are the smaller half of the story. The repo also ships the evaluation harness that tried to kill them, and publishes the result: no skill in the pack is proven to make the model more accurate.

The finding

Every skill went through what the repo called an Elevate-or-Kill pipeline: length-controlled (skill prompt vs a same-length placebo), replication-gated (an apparent win must survive an independent rerun), with objective SWE-bench localization evals for the debugging skills and routing evals for discoverability and false positives.

• Zero skills hold a robust, replicated ELEVATE verdict. The audit (analysis/AUDIT.md, JSON registry wins on disagreement) lists Elevate 0 / AUTO-RETAIN 0, global validity provisional, disposition no_automatic_elevation.
• The August refresh moved the numbers. The July write-up of thinking-scientific-method (hypothesis-differential debugging) was +5.3pp, p=0.061, n=150 on the primary run, with a separate replication at +8.0pp, p=0.001. The current README and audit give the best directional row as +4.0pp, below the pack's own +5pp utility margin, and call the July estimates provisional because of scoring, placebo and archive defects. The older figures are kept here as what the July version of the repo reported (reasoning-scaffolds cites them).
• The pack was cut, not defended. 11 of 39 skills were deleted after their mechanisms were absorbed into others (archetypes, feedback loops and leverage points into systems, for example); the 28 survivors were lean-rewritten and are all manual-only, with no automatic invocation. The README: "The evidence supports structured procedures, not a guaranteed accuracy gain."
• The second round of evals never ran. portfolio-v1 and workflow-v1 both record no_run with zero model calls; the audit says a zero-call result is not a measured null. The workflow study failed a pre-run power check on 24 cases, so the workflow machinery is marked for deletion.
• A cleanup incident is on the record. The recovery ledger says 14 frozen historical result artifacts without raw-response archives were deleted by mistake; none is referenced by the current claim registry, and none may back a public claim unless the exact bytes are restored.

The length control is what makes this bite. If a skill's benefit were mostly "more tokens of deliberation before answering," a placebo of equal length would capture it — and apparently does. What the design cannot see is anything the accuracy metric doesn't measure: consistency across runs, the legibility of the reasoning to a human reviewer, or steering on open-ended work with no scoreable answer. A negative on multiple-choice-style accuracy is not a negative on usefulness, and the author doesn't claim otherwise.

Why it matters to this spoke

This is the first source here that measured whether a skill pack works, and it's the author's own pack.

The spoke has accumulated skill packs on assertion alone — cybersecurity-skills (817 skills), pm-skills-arsenal, agentic-seo-skill, superpowers, compound-engineering-plugin — every one documented by its README's claim of what it improves. The standing Benchmarks open question has been waiting for a neutral measurement since the spoke was split out. cc-thinking-skills doesn't close it (n is small, single-author, one harness, and it measures thinking skills rather than capability skills), but it's the first entry that ran the experiment and reported against its own interest.

It also lands squarely on the does structure substitute for capability question, at the layer where the substitution claim is cheapest to make. agentskills-spec's progressive disclosure is a real token-budget mechanism, measurable and mechanical. "Reasoning better because you were handed a framework" is a different kind of claim, and it's the one that failed to replicate. See reasoning-scaffolds for the distinction between skills that grant an agent a capability and skills that prescribe how it thinks.

The Cognition "trust no eval" stance (cognition-fable5-through-the-night) is the useful contrast: both refuse to take a benchmark at face value, but from opposite ends — Cognition distrusts public benchmarks and builds a proprietary one for shippability; Boudreaux builds a public one and lets it return a verdict he'd rather not have.

The rest of the repo

• thinking-model-router as front door — a meta-skill whose job is routing to the other 27, the same advertise-then-load shape agentskills-spec describes, applied within a single pack. Its own routing evals score discoverability and false-positive rate, i.e. whether a skill fires when it shouldn't.
• QA tooling — validate-skills.js (frontmatter and required-section lint, per-skill quality score), enhance-skill.js (improvement suggestions), generate-improvement-prompt.js (emits a prompt for Claude to rewrite a skill). Skills being maintained by the agent that runs them is the self-improving-agents loop in its supervised, human-in-the-loop form.
• Provenance of the frameworks — Munger (mental models), Meadows (systems thinking), Kahneman (dual-process), Goldratt (constraints), Altshuller (TRIZ), Boyd (OODA). Established, cited, and out of scope for this wiki beyond the packaging.

What the refresh adds

The repo stopped at 2026-08-07 (four commits that week, the last a README refresh co-authored with Claude Opus 4.8). The first ingest read a harness that returned a null on a 39-skill pack. This read finds the author acting on the null: shrinking the pack, removing auto-invocation, and refusing to label anything proven. It also finds the follow-up studies unrun, so the spoke's only ablation of a comparable pack is still the July one, now flagged by its own author as defective in scoring and placebo handling. That weakens the +5.3pp figure more than it strengthens the null, and the open question of whether thinking skills help stays open.

Related

reasoning-scaffolds · agentskills-spec · agent-skills · superpowers · cybersecurity-skills · pm-skills-arsenal · self-improving-agents · cognition-fable5-through-the-night · auto-company · tj-boudreaux