WiCER — Wiki-memory Compile, Evaluate, Refine

Juan M. Huerta (juan-huerta, Zinnia Tech Solutions, Greenwich CT), arXiv 2605.07068, submitted 8 May 2026. T1: a primary study with a stated method, a public dataset and released numbers. The first source in this corpus that measures what the compilation step costs — and it costs more than anyone here had assumed.

The gap it names

The llm-wiki pattern rests on a step nobody had graded. Raw documents go in, an LLM distils them into wiki pages, and the wiki is thereafter served to models through a warm KV cache — sub-second context with no retrieval to fail. Huerta calls the unexamined middle the compilation gap: "distilling raw documents into a wiki without catastrophically discarding critical facts." Everything this spoke holds about the pattern — llm-wiki-gist, wiki-memory, gbrain — argues the merits of the compiled artifact. None of them asks what fell out during compilation.

Method

• Diagnostic probes. One QA pair per source document, drawn from the original collection. A probe is a question the raw corpus can answer; if the compiled wiki cannot, the compiler dropped something.
• The loop (Algorithm 1), borrowed in shape from CEGAR — counterexample-guided abstraction refinement, out of formal verification. Compile blind → score the wiki against the probes with an LLM judge → take every failure → extract the specific facts those questions needed → recompile with those facts pinned → repeat, up to T iterations, stopping when failures hit zero or the improvement falls under 10%.
• Scoring. Claude Sonnet as judge on a 1–5 rubric, 5 = perfect, 1 = wrong or irrelevant. A 1 is the paper's "catastrophic failure" — not a thin answer, an answer that misses the point. Judge–human agreement checked on 100 samples, r = 0.94.
• Data. RepLiQA: 17 topics (company policy, cybersecurity, local news, others), 80 documents and 400 QA pairs per topic, 6,800 questions; 15 topics carry the full baseline comparison.

What it found

Blind compilation destroys the majority of what it is asked to preserve. More than half the probes come back scored 1 — the compiled wiki does not merely summarise the fact away, it answers wrongly. That is the number this corpus did not have.

The repair is cheap and partial. One or two refinement passes recover about 80% of the lost quality and cut catastrophic failures by 55% relative. Most topics (10 of 17) peak at iteration 2; seven peak at 1 or earlier, so the loop is not a knob you keep turning. And targeted diagnosis is what does the work: pinning the specific facts a failed probe needed is worth +0.95, against +0.16 for pinning facts generically.

A compiled wiki does not reach the raw documents it came from. Best WiCER (3.24) still sits under full-context on the raw corpus (3.47) and under RAG (3.64). Compilation buys latency, not fidelity.

The finding that contradicts this spoke

wiki-memory and llm-wiki both carry Harrison Chase's contrast: RAG re-retrieves raw chunks per query, a wiki precomputes the synthesis, so the wiki is the better shape. This paper reports the reverse, and it is careful about where:

• On 30 curated articles (~67k tokens, 70% of a 96k window) full context beats RAG 4.38 vs 4.08, at 7.3× faster time to first token — the case the pattern's advocates describe.
• On 80 raw documents per topic (55–95k tokens, 57–99% window fill) it flips: "RAG consistently outperforms full-context: RAG mean is 3.64" against 3.47. Huerta attributes it to attention dilution.

So the pattern's advantage is real and it is conditional on the corpus already being small and curated. Push raw volume up and retrieval wins on quality while full context keeps the latency win. That crossover is the paper's actual motivation for compiling at all, and it is not how this spoke has been stating the RAG contrast. Recorded as a contradiction in synthesis rather than resolved.

Limits the paper states

One model (Llama 3.1 8B), one machine (Apple M4 Pro), greedy decoding at T=0 — so no variance across runs. RAG runs fixed-size chunks with no reranking, which flatters nothing but is a weak baseline to lose to. One of the 17 topics gets no improvement from WiCER at all. Human validation of the judge covers 100 samples.

Why it matters here

Every wiki in this hub is a blind compilation. Nothing in the hub's ingest gate is a probe: the rung-0 checkers grade links, tiers, freshness and arguments — properties of the pages that exist — and none of them can see a fact that never made it out of a source. check:arguments was the closest thing, and it grades sentences the compiler wrote. WiCER's probe is the missing instrument: a question the raw source answers, asked of the wiki. See synthesis — this is now an open question about our own method, not only about other people's.

Related

llm-wiki · wiki-memory · wikis-over-engineered-compiler · wiki-compiler · retrieval-augmented-generation · gbrain · llm-wiki-gist · juan-huerta · synthesis