cc-thinking-skills

Claude Code Thinking Skills (TJ Boudreaux, MIT, ~822 stars / 125 forks as of 2026-07-22) is a pack of 39 mental-model and critical-thinking frameworks shipped as agent skills — first-principles, inversion, pre-mortem, Bayesian updating, Cynefin, theory of constraints, OODA, TRIZ, Fermi estimation, red-teaming — plus a thinking-model-router meta-skill that reads a problem and points at the right framework. Installable through the Claude Code plugin marketplace or by copying skills/* into ~/.claude/skills/.

The skills are the smaller half of the story. The repo also ships the evaluation harness that tried to kill them, and publishes the result: no skill in the pack is proven to make the model more accurate.

The finding

Every skill went through what the repo calls an Elevate-or-Kill pipeline: length-controlled (skill prompt vs a same-length placebo), replication-gated (an apparent win must survive an independent rerun), with objective SWE-bench localization evals for the debugging skills and routing evals for discoverability and false positives.

• Headline: zero skills hold a robust, replicated ELEVATE verdict.
• Closest candidate: thinking-scientific-method (hypothesis-differential debugging) — fresh primary run +5.3pp, p=0.061, n=150, which misses the p<0.05 gate. A separate replication came in at +8.0pp, p=0.001, but by the pipeline’s own rule a significant replication can’t rescue a failed primary, so the verdict stands at DIRECTIONAL-NOT-REPLICATED. A larger pre-registered study is listed as future work.
• All 39 skills ship anyway. The README’s reading: treat them as “well-grounded structured-reasoning scaffolds, not as a guaranteed accuracy boost” — “the frameworks are real and useful; the empirical proof of a model-accuracy lift is not yet there, and we say so.”

The length control is what makes this bite. If a skill’s benefit were mostly “more tokens of deliberation before answering,” a placebo of equal length would capture it — and apparently does. What the design cannot see is anything the accuracy metric doesn’t measure: consistency across runs, the legibility of the reasoning to a human reviewer, or steering on open-ended work with no scoreable answer. A negative on multiple-choice-style accuracy is not a negative on usefulness, and the author doesn’t claim otherwise.

Why it matters to this spoke

This is the first source here that measured whether a skill pack works, and it’s the author’s own pack.

The spoke has accumulated skill packs on assertion alone — cybersecurity-skills (817 skills), pm-skills-arsenal, agentic-seo-skill, superpowers, compound-engineering-plugin — every one documented by its README’s claim of what it improves. The standing Benchmarks open question has been waiting for a neutral measurement since the spoke was split out. cc-thinking-skills doesn’t close it (n is small, single-author, one harness, and it measures thinking skills rather than capability skills), but it’s the first entry that ran the experiment and reported against its own interest.

It also lands squarely on the does structure substitute for capability question, at the layer where the substitution claim is cheapest to make. agentskills-spec‘s progressive disclosure is a real token-budget mechanism, measurable and mechanical. “Reasoning better because you were handed a framework” is a different kind of claim, and it’s the one that failed to replicate. See reasoning-scaffolds for the distinction between skills that grant an agent a capability and skills that prescribe how it thinks.

The Cognition “trust no eval” stance (cognition-fable5-through-the-night) is the useful contrast: both refuse to take a benchmark at face value, but from opposite ends — Cognition distrusts public benchmarks and builds a proprietary one for shippability; Boudreaux builds a public one and lets it return a verdict he’d rather not have.

The rest of the repo

• thinking-model-router as front door — a meta-skill whose job is routing to the other 38, the same advertise-then-load shape agentskills-spec describes, applied within a single pack. Its own routing evals score discoverability and false-positive rate, i.e. whether a skill fires when it shouldn’t.
• QA tooling — validate-skills.js (frontmatter and required-section lint, per-skill quality score), enhance-skill.js (improvement suggestions), generate-improvement-prompt.js (emits a prompt for Claude to rewrite a skill). Skills being maintained by the agent that runs them is the self-improving-agents loop in its supervised, human-in-the-loop form.
• Provenance of the frameworks — Munger (mental models), Meadows (systems thinking), Kahneman (dual-process), Goldratt (constraints), Altshuller (TRIZ), Boyd (OODA). Established, cited, and out of scope for this wiki beyond the packaging.

Related

reasoning-scaffolds · agentskills-spec · agent-skills · superpowers · cybersecurity-skills · pm-skills-arsenal · self-improving-agents · cognition-fable5-through-the-night · auto-company · tj-boudreaux