Wiki compiler (deterministic pipeline)

The deterministic counterpart to the LLM-wiki agent loop: building and maintaining a linked markdown wiki with a plain program (a “compiler”) rather than recursive LLM calls. Named and argued in wikis-over-engineered-compiler. The move rests on a division of labor — separate the mechanical 90% of wiki-work from the semantic 10%:

• Deterministic code owns text parsing, restructuring, cross-referencing (lexical/exact-name matching), graph traversal, and structural validation — “work where the outputs are deterministic; they don’t move.”
• The LLM is reserved for genuine semantic judgment — recognizing that two differently-worded phrases mean the same thing (the link a lexical matcher can’t make).

The pipeline shape

A typical compiler is a short, dependency-free pass: extract metadata (tolerant regex) → build the cross-reference graph (a word-indexed phrase matcher, not pairwise compares) → rewrite each page, regenerating owned sections (Metadata / Related / Referenced By / Body) while preserving hand-written ## Notes → lint for broken links and orphans. Its defining properties are the ones an agent lacks: byte-identical, OS-independent output on repeated runs, near-instant speed (sub-second over thousands of files), and no token cost.

The trade

The guarantee comes at the cost of paraphrastic linking: exact-name matching won’t connect “gradient descent” and “the optimization step.” So the compiler and the agent aren’t rivals so much as the two ends of a deterministic ↔ probabilistic axis — the compiler for structure you want guaranteed, the LLM for meaning you can only infer. It sits alongside open-knowledge-format (git-diffable typed markdown, no probabilistic edges) and the human-in-the-loop sift-kg on the “distrust the model for structure” side of the cluster, opposite gbrain‘s unattended reasoning daemon.

Reflexively, most LLM-maintained wikis (this hub among them) are already hybrids: an LLM does semantic ingest and synthesis, while deterministic scripts do the link/count/orphan validation — which is exactly the split this concept names.

Related

llm-wiki · wikis-over-engineered-compiler · open-knowledge-format · sift-kg · gbrain · knowledge-graph