LLM×MapReduce

A divide-and-conquer framework for long-to-long generation — producing a long document from inputs too large to fit any context window. Apache-2.0, ~872★, maintained jointly by THUNLP (Tsinghua NLP), OpenBMB and AI9STARS. It powers SurveyGO, a system that writes survey articles from a literature corpus.

The framing the authors use is that the field has worked hard on long-input understanding and barely at all on the case where the output is also long: “generating long texts from extremely long resources remains relatively underexplored.”

The mechanism

MapReduce, applied to context. Split the corpus, process fragments independently, then combine — but the combination is where the design sits, because naive concatenation loses the cross-fragment structure a survey needs.

• V1 handles long-sequence understanding with a structured information protocol and confidence calibration, so fragments carry enough metadata to be merged rather than stacked.
• V2 adds entropy-driven convolutional test-time scaling: repeated passes that build higher-level understanding from local fragments, the way a convolutional network builds features from pixels. Reported on SurveyEval at 95.50% precision / 95.80% recall against vanilla baselines at 25.48% / 26.46%.

Treat that margin with the usual caution — it is the authors’ own evaluation of their own system on their own dataset, which is the condition graphrag-bench exists to correct for in the adjacent cluster. The size of the gap (roughly 4×) is more likely to say something about how weak the vanilla baseline is at this task than about how close to solved it is.

Why it lands in this wiki rather than next door

This is the third distinct answer the corpus holds to “the corpus does not fit, now what.” retrieval-augmented-generation retrieves the relevant slice. graphrag builds a structure first and traverses it. This one processes everything and folds the results, which is the option neither of the others takes, and the only one whose output is designed to be long.

It also reads on long-context-vs-chunking from the other end. That page asks when a long context window is worth its cost for a reader. LLM×MapReduce says the question stops applying past a certain input size — no window is long enough for a literature corpus, so the architecture has to be map-reduce whatever the window costs.

A metric this wiki went looking for today and could not find

V2 defines claim density: unique claims over total extracted claims, after intra-group and cross-group deduplication. That is a measurement of claim-level redundancy, and this corpus spent a research pass on 2026-08-08 hunting for exactly that shape of number for claim-shaped graph nodes and finding it implemented everywhere and scored nowhere.

It does not close that gap, and the reason is worth being precise about. Density here scores a generated article — how much of the writing is non-redundant — not a persistent graph whose nodes accumulate over time. The deduplication is a pipeline step in service of the output, and the paper reports the ratio rather than the merge quality: nothing says how many of those merges were right. So it is the nearest thing the corpus has to the missing measurement and still the wrong side of it.

Worth noting the pattern, since it has now recurred: claim-level deduplication keeps appearing as machinery and never as a result. Three sources in one day, same shape.

Related

long-context-vs-chunking · retrieval-augmented-generation · graphrag · entity-resolution · knowledge-graph · graphrag-bench · synthesis