“LLM Wikis Are Over-Engineered — I Replaced Mine With a Pure Python Compiler” (TDS)

A Towards Data Science piece arguing that the LLM-wiki pattern points a probabilistic tool at a deterministic job. The author started from Karpathy‘s agent-loop approach (llm-wiki-gist) — recursive LLM calls + embeddings to organize local markdown — and abandoned it on realizing he was “paying tokens to reorganize text I already owned.” The replacement is a deterministic Python compiler; the reframe is a division of labor between mechanical and semantic work.

The critique

Three costs of doing wiki-bookkeeping with an agent, all traced to using reasoning for parsing:

• Cost — token spend on organizational work scales badly with corpus size.
• Latency — network round-trips for purely mechanical restructuring.
• Non-determinism — running the same folder twice produced different link structures (“slightly different judgment calls both times”). The headline line: “An agent decides what your wiki might look like. A compiler guarantees what it must look like.”

The compiler (stdlib-only, four stages)

1. Regex metadata extractor — tolerant of messy/inconsistent headers and optional fields (scans wherever they appear rather than enforcing position).
2. Graph builder — a word-indexed phrase matcher for cross-references instead of pairwise regex; the optimization cut 5,000-file processing from 107 s to under 1 s (“the algorithm mattered more than avoiding API calls”).
3. Section-aware rewriter — rebuilds compiler-owned sections (Metadata / Related / Referenced By / Body) but preserves anything under ## Notes (hand-written, kept across recompiles).
4. Linter — broken-link + orphan checks (he found a bug double-counting incoming links from the generated “Referenced By” section).

Links are [[Entity Name]]; output is byte-identical across runs and OSes (orphan counts stable every time). Recompile: 171 ms / 1.8 s / 12.4 s at 100 / 1,000 / 5,000 files.

Where the LLM stays

The author keeps the LLM for the genuinely semantic 10% — recognizing that “gradient descent” and “the optimization step” mean the same thing (his compiler’s exact-name matching can’t). “Reserve LLMs for the … part requiring actual semantic understanding … the 90% that is mechanical restructuring … needs a parser and validator.” Thesis: “If your input is deterministic, your pipeline should be too.”

Why it matters here

It’s the sharpest counter-argument to the llm-wiki thesis that “LLMs make maintenance near-free, so the wiki stays maintained”: true, says the author, but the mechanical maintenance shouldn’t be probabilistic at all. It draws a new axis for the cluster — deterministic compiler vs. reasoning agent — orthogonal to the augment→automate one. And it’s reflexively pointed at wikis built exactly this way (this hub included), which already run deterministic link/count/orphan checks beside the LLM ingest — a hybrid that partly embodies the author’s split (see synthesis).

Tier T3 — a single practitioner’s opinion piece; benchmarks are self-reported (though reproducible in spirit — byte-identical output is a checkable claim). No independent evaluation; the author is candid about the limit (exact-name matching misses paraphrastic links).

Related

wiki-compiler · llm-wiki · llm-wiki-gist · open-knowledge-format · sift-kg · gbrain · andrej-karpathy