sift-kg

A CLI tool that turns a folder of documents into a knowledge-graph via LLM extraction, by juan-ceresa (MIT-licensed). Drop in PDFs/articles/records and it produces a browsable property graph of entities and typed relationships, with every edge carrying a confidence score and a pointer back to the source passage. It positions itself two ways at once: a PKM alternative to hand-building a wiki (“structure emerges from the documents”), and an “AI second brain” — persistent, queryable memory for AI agents (JSON API).

The pipeline

Extract text from 75+ formats (Kreuzberg engine; OCR via Tesseract/EasyOCR/PaddleOCR).
Schema — either let the LLM discover an entity/relation schema from the corpus, or apply a predefined domain (general, osint, academic, or schema-free).
Extract entities + relations via LLM (LiteLLM → OpenAI/Anthropic/Mistral/Ollama): typed nodes (PERSON, ORGANIZATION, …) and typed edges (EMPLOYED_BY, OWNS, …), each with a confidence score and source provenance.
Build the graph in NetworkX (a property graph; JSON is the native store).
Resolve + review + apply — deduplicate entities (see below).
View / search / export / narrate — interactive browser view, CLI lookup, export to GraphML/GEXF/CSV/SQLite (Gephi/Cytoscape), or an LLM-generated prose summary.

The distinctive bit — human-approved entity resolution

Deduplication runs in three layers, and the merge decision is gated on a human:

Automatic pre-dedup — Unicode normalization, title stripping, fuzzy match (SemHash); optional embedding clustering (sentence-transformers + scikit-learn).
LLM proposal — batched entity comparison; the model proposes merges, stored as DRAFT.
Human review — interactive CLI or YAML editing marks each proposal CONFIRMED / REJECTED; only confirmed merges are applied. “No merges occur without explicit approval.”

See entity-resolution for how the wiki’s other builders answer the same problem — including rahulnyk-knowledge-graph, which ships without deduplication and lists it as its top open request.

This is the design point that places sift-kg on the wiki’s augment→automate axis. Where gbrain dedups and re-cites unattended in a “dream cycle” while you sleep (the automate pole), sift-kg deliberately keeps the human in the merge loop (the Engelbart augment pole) — a direct counter-answer to the “who maintains the trails?” question the synthesis tracks: here, you approve the merges, the LLM only proposes.

How it builds the graph — vs the wiki’s other strategies

sift-kg is the packaged, domain-agnostic instance of the two-pass LLM extraction ontologies-knowledge-graphs-ai (Mysore) describes: draft a schema, then populate a graph constrained by it — its predefined domain packs and schema-discovery mode are exactly the schema-as-guardrail idea (the ontology bounds what the model may assert). It sits among the wiki’s graph-building strategies: unlike gbrain (zero-LLM, wikilink pattern-matching) and llm-wiki-agent (deterministic + inferred passes over markdown), sift-kg extracts from arbitrary unstructured documents with the LLM doing the entity/relation lifting — closer to Mysore’s GraphBaby than to the wikilink-native tools, but shipped as a general CLI rather than a demo. It stops at extraction, though: the retrieval half — community detection and global-query answering — is left to the user. graphrag-rs is the sibling that closes that half (it runs GraphRAG‘s Leiden communities over the extracted graph), making sift-kg the graph-builder and graphrag-rs the graph-builder+querier.

Where it sits on the formal↔informal axis

Its graph is a property graph (NetworkX) + JSON, not RDF triples — so it stays on the non-semantic-web side, the sibling knowledge-representation-wiki’s RDF/OWL tradition is a cross-spoke contrast, not its home. But unlike open-knowledge-format (which declines typed edges entirely, leaving relationships as prose for an LLM to read), sift-kg keeps typed, scored edges — just in a property graph rather than a triplestore. So it’s a middle position: more structure than OKF, less formalism than RDF/OWL.

As agent memory

The “second brain for agents” framing (structured, queryable topology via JSON API) is the agent-memory-knowledge-graphs use case — but sift-kg extracts a static snapshot of a document set, with no time axis, so it’s the non-temporal sibling of the temporal-knowledge-graph / graphiti approach: good for mapping a fixed corpus, not for tracking facts that change over time.

Tier note (T3)

Project README / self-description of an MIT open-source tool — concrete (specific pipeline, named dependencies, real CLI) but vendor-authored and not independently evaluated. No benchmark or third-party accuracy comparison; extraction quality is asserted, not measured.

knowledge-graph · graphrag · graphrag-rs · ontologies-knowledge-graphs-ai · ontology · gbrain · llm-wiki-agent · agent-memory-knowledge-graphs · open-knowledge-format · tools-for-thought · juan-ceresa