When to use Graphs in RAG (GraphRAG-Bench)

A neutral third-party benchmark of graph-based RAG (Xiang, Wu, Q. Zhang, S. Chen, Hong, X. Huang, Su; arXiv:2506.05690, accepted ICLR 2026 per the project repo). The authors propose no method of their own — they build a benchmark to answer when graphs actually help. This is the source the RAG thread’s open question had been waiting for: every prior data point here came from an advocate (gbrain) or a method’s own paper (GraphRAG, leanrag), all author-run and LLM-judged.

The headline

“GraphRAG frequently underperforms vanilla RAG on many real-world tasks.” Graphs are not a free upgrade; their benefit is task-dependent, and they carry a steep token-cost premium.

What it tests

GraphRAG systems: Microsoft GraphRAG (local & global), LightRAG, HippoRAG & HippoRAG2, RAPTOR (a summary tree, not an entity graph), Fast-GraphRAG, Lazy-GraphRAG.
Baselines: basic vector RAG, with and without reranking.
Corpora: a Novel set (pre-20th-century Project Gutenberg literature) and a Medical set (NCCN clinical guidelines).
Four difficulty levels: fact retrieval → complex (multi-hop) reasoning → contextual summarization → creative generation. The whole pipeline is scored, from graph construction through retrieval to generation.

When graphs win, when they lose

Vanilla RAG wins on simple fact retrieval — discrete facts sit in single passages, and the graph “introduces logically relevant but redundant information.” Reported accuracy: basic RAG ~61–65% vs GraphRAG methods 49–60%.
Graphs win on multi-hop reasoning and summarization — bridging interconnected concepts and synthesizing fragmented knowledge. Complex-reasoning accuracy: RAG ~43% vs HippoRAG2 ~54%; GraphRAG hits 87.9–90.9% recall on complex tasks (though sometimes at worse context relevance than vector RAG).

The cost axis (matches GraphRAG’s own finding, from the skeptical side)

Per-query token cost varies by 1–2 orders of magnitude: vanilla RAG ~879–954 tokens; HippoRAG2 ≈10³ (the most efficient graph method); LightRAG ≈10⁴; Microsoft GraphRAG global up to ~4×10⁴. So the global map-reduce mode from-local-to-global-graphrag sells for global sensemaking is also the most expensive by far — useful when the question is genuinely global, wasteful when it isn’t.

Why it matters here

It closes the open question the KG/RAG thread carried (“a neutral third-party benchmark comparing the approaches”) and reframes the cluster’s advocate narrative: graphs are a task-conditional tool, not a blanket win over vector RAG. It corroborates the cost story from from-local-to-global-graphrag while contradicting the broad “graphs beat vector RAG” framing of gbrain and the method papers — a tension now flagged in synthesis. Author institutions not listed in the abstract; authors noted inline (thin).

The neutrality claim needs scoping (added 2026-08-03)

The sentence above — “the authors propose no method of their own” — was true of this paper and is no longer true of its authors. memgraphrag (KDD 2026) is a GraphRAG method from largely the same group: this benchmark’s authors are Zhishang Xiang, Chuanjie Wu, Qinggang Zhang, Shengyuan Chen, Zijin Hong, Xiao Huang, Jinsong Su; MemGraphRAG’s are Chuanjie Wu, Zhishang Xiang, Yunbo Tang, Zerui Chen, Qinggang Zhang, Jinsong Su. Four names shared, including both of MemGraphRAG’s equal-contribution first authors and its corresponding author. MemGraphRAG evaluates on G-Bench(Medical) and G-Bench(Novel) and tops both.

This does not retract anything on this page. The benchmark was published first, its diagnosis has been taken up by others, and building a benchmark before building a method is the right order. What changes is scope: “neutral third party” describes this benchmark’s relationship to the systems it originally tested, not to MemGraphRAG’s results on it. Read those particular numbers as a method paper’s self-evaluation, and keep the benchmark’s independence claim for everyone else.

There is a second-order point worth keeping. This benchmark’s value to the wiki was that it came from outside the advocacy — and the same team then produced the advocacy. The pattern to watch is not dishonesty but drift: a group that diagnoses a field’s weakness is unusually well placed to build the thing that fixes it, and unusually poorly placed to score it.

graphrag · from-local-to-global-graphrag · leanrag · graphrag-rs · retrieval-augmented-generation · knowledge-graph · hybrid-retrieval-rag · gbrain · memgraphrag · graph-construction-quality