Spokes.wiki Search About

Synthesis — Analytical Databases

The evolving thesis. Spun out of the hub _inbox analytical-databases cluster on 2026-06-18 at the human's direction — a deliberate early spin-out at two sources (below the usual ≥3), both ClickHouse-vs-DuckDB comparisons. So the founding corpus is narrow: one axis (embedded vs distributed OLAP), seen once through a vendor lens and once through an independent practitioner lens. The domain is much larger than that, and the thesis below is a starting frame to test as warehouses, real-time platforms, and more engines arrive.

Current thesis

Analytical (OLAP) databases sit on an embedded↔distributed spectrum, and the right answer is usually a position on it — often more than one position at once.

Added 2026-06-30: the warehouse end is now sourced (Snowflake, T1). It reframes the founding "use both" finding: PostHog reached for embedded DuckDB because a coupled engine didn't fit per-tenant warehousing — disaggregation is the other escape hatch from the same coupling problem.

The sharpest finding so far: the "vs" framing is partly false. The independent PostHog account is the load-bearing source — at production scale they run both, for different layers: ClickHouse for their own product event-analytics, and DuckDB (wrapped in Duckgres on DuckLake + a Postgres catalog) for a per-customer data-warehouse product that had to scale independently, because self-managed ClickHouse "would not work for data warehousing for all of our customers." So the practitioner question isn't "which wins" but "which engine for which layer of the stack" — embedded per-tenant warehousing vs centralized high-scale event analytics.

Open questions

Growth edges

Ranked; each names the kind of source that would close it (see ../QUALITY.md → Growth edges).

  1. The real-time serving family. Half closed 2026-08-15 (quality cycle) — druid-real-time-analytical-data-store (Yang et al., SIGMOD 2014, T1) supplies the family's architecture: node-type separation, in-memory-then-segment ingestion, timestamp-mandatory partitioning, and the deliberate absence of joins. The shape turned out to be freshness plus concurrency, not pre-aggregation, which is how this edge had described it. Successor: Druid is one design and one company's production cluster from 2014. Pinot and StarRocks remain unsourced, and nothing compares the three or measures them against clickhouse, which is the engine practitioners actually weigh them against. — needs: the Pinot or StarRocks paper, or any benchmark putting a real-time engine beside a general columnar one. Advanced 2026-08-16 (research pass), and the half that closed is not the half that was asked for. pinot-realtime-olap (Im et al., SIGMOD 2018, T1) gives the spoke its second real-time design (apache-pinot) and the family's first head-to-head — Pinot against Druid on nine stated hosts, production data, production queries. So "one design" is answered and "nothing compares them" is half-answered. StarRocks was hunted and refused: it has no paper. Every StarRocks-vs-ClickHouse comparison in reach is published by StarRocks or its commercial arm CelerData (SSB 2.2×, TPC-H 3–5×), and the nearest counterweight is Tinybird's, which sells managed ClickHouse. Both sides sell the winner. Needs now: a real-time engine measured against clickhouse by someone selling neither — an academic benchmark, or an operator publishing a migration with numbers. The direct comparison the edge was written for is still missing, and the market's own literature cannot supply it. Hunted again 2026-08-30 (research pass) — the operator half arrived and the ClickHouse half did not. realtime-data-infrastructure-uber (Fu & Soman, SIGMOD 2021, T1) is exactly the operator publishing at scale the edge names: petabytes and trillions of messages per day, Pinot from dozens of GB to several hundred TB and from a few hundred QPS to tens of thousands in two years, against a p99-under-one-second requirement. It measures Pinot against Elasticsearch (4× the memory, 8× the disk, 2–4× the query latency) and against Druid not at all — ClickHouse appears once, in related work. So the edge stands as written. Successor edge 1a below records what the hunt found instead. Hunted 2026-09-17 — nothing ≥ bar. StarRocks' SSB page (2.2× vs ClickHouse, 8.9× vs Druid) is the vendor's own; ByteHouse (ByteDance, arXiv 2602.08226, 2026) is a real paper but benchmarks against unnamed "existing systems" on internal workloads. No Pinot or StarRocks paper. Not eligible again before 2026-10-01. 1a. The Pinot-over-Druid claim has one pool of sources, and one of them is a blog. Discovered 2026-08-30 while hunting edge 1. Uber, the largest Pinot operator, supports its Druid rejection in a SIGMOD paper by citing "recent studies done outside of Uber" — reference [47] of which is a Medium post. This spoke's own Pinot-over-Druid material (pinot-realtime-olap, realtime-data-infrastructure-uber) therefore traces back to a narrower base than two independent T1 sources suggests. — needs: a Pinot-vs-Druid measurement by a party affiliated with neither, on stated hardware. T1/T2.
  2. The rest of the disaggregated family. Snowflake is sourced from its SIGMOD paper; BigQuery (Dremel/Colossus) and Databricks (lakehouse) are the same family and unsourced. — needs: the Dremel or lakehouse paper (T1).
  3. Engine versus packaging, measured. tinybird (serving) and jitsu (ingestion) both argue the packaging around the engine is the real product, and neither supplies a number. — needs: any measurement separating engine performance from the managed layer. Sharpened 2026-10-07, promoted from the dream ledger. Tinybird names the serving side as where ClickHouse gets hard to run. But ClickHouse writes each insert as a new immutable part that a background process has to merge later — frequent small inserts build a merge backlog and can trip "too many parts" errors, a write-path cost rather than a serving one. Jitsu's Bulker batches before loading, which looks like the same problem answered from the other end of the pipe. If the write path is where the operational cost concentrates, Tinybird's own framing is naming the wrong half of the gap. — needs now: a load test of an ingestion pipeline into self-hosted ClickHouse at several batch intervals, measuring part count, merge backlog and insert errors against batch size.

Coverage edges (added 2026-08-08, at the curator's request for a wider backlog). These widen what the spoke covers instead of answering an open question above; one ordinary solid source closes any.

  1. The file formats everything rests on. CLOSED 2026-08-09 (research pass) — parquet, orc and apache-arrow from their own specifications, plus columnar-format-evaluation (Zeng et al., VLDB 2023) as the benchmark, all T1. Successor: what replaces them. The paper says the standard formats encode early-2010s assumptions; several projects (BtrBlocks, Lance, Vortex, Nimble) claim to be the answer and the spoke holds none of them. — needs: a next-generation format's design paper, T1/T2. CLOSED 2026-08-22 (research pass) by btrblocks (Kuschewski, Sauerwein, Alhomssi & Leis, SIGMOD 2023, T1). Successor, edge 8: one of the four, from one group, benchmarked by its own authors against their own Parquet harness. Lance, Vortex and Nimble are unsourced and nothing compares any two next-generation formats to each other. — needs: a second format's design paper, or any third-party comparison of them.
  2. Open table formats. Iceberg, Delta Lake and Hudi are where the lakehouse claim is actually made, and storage-compute-disaggregation has no counterpart page for the metadata layer that makes the shared storage writable. — needs: a specification or design paper.
  3. A query engine that owns no storage. Trino and Presto separate execution from the store, the clean case for the disaggregation axis. — needs: the Presto SIGMOD paper or Trino's own docs.
  4. What people put in these engines. dbt and dimensional modelling (Kimball) shape more analytical workloads than engine choice does, and the spoke covers only engines. — needs: dbt's documentation or a modelling text.
  5. The other next-generation formats. (new 2026-08-22, successor to edge 4) btrblocks is one of four candidates columnar-format-evaluation pointed at; Lance, Vortex and Nimble are unsourced, and no comparison of any two of them exists here. The BtrBlocks numbers are also its own authors' against their own Parquet harness, unreplicated. — needs: a second next-generation format's design paper, or a third-party comparison across them.

Two designs in one family, and one of them tuned the other (2026-08-16)

pinot-realtime-olap arrives eleven months of spoke-time after druid-real-time-analytical-data-store and four years after it in publication time, from linkedin rather than Metamarkets, and it makes the real-time family legible in a way one paper could not.

What the two agree on is more than the vocabulary: immutable segments, node-type separation (broker / server / coordinator-controller), streaming ingestion made queryable before columnar conversion, and a hard commitment to concurrency at sub-second latency. That is the family's shape, and it is now attested twice by two teams who did not collaborate.

What they disagree about is where the speed comes from, and the disagreement is measurable. Druid indexes every dimension column; Pinot indexes selectively and sorts rows physically on the identifier queries filter by. On one production dataset that difference shows up as 1.2 TB against 300 GB for the same workload. Pinot's own measurement credits layout — physical ordering — over index structure on its highest-throughput case, and credits its star-tree pre-aggregation on the ad-hoc case. Without any indexes at all, the two engines perform comparably, which is a quiet way of saying the architecture is shared and the tuning is the product.

The methodological point is the one to keep. Druid's paper benchmarked against MySQL/MyISAM, declining to tune a rival column store; Pinot's paper does tune the rival, and the rival is Druid. Neither escapes self-interest, and the corpus now holds both directions of the same bias — the paper that refuses to compare and the paper that compares on its own terms. That is as close to a controlled read as this literature currently offers, and it is still not close.

What is still absent is the comparison practitioners actually make: a real-time engine against clickhouse, measured by someone selling neither. The hunt for StarRocks found no paper at all and a market where every published benchmark belongs to a vendor on one side or the other — StarRocks and CelerData claiming 2.2× on SSB and 3–5× on TPC-H, Tinybird answering while selling managed ClickHouse. A family this commercially contested may simply not produce a neutral number.

A compression ratio is the wrong number for a networked scan (added 2026-08-22)

btrblocks closes the format-successor edge and lands on something this spoke has been quoting carelessly. Compression ratios are how parquet, orc and every vendor page argue, and for a scan pulled across a network they measure the wrong thing.

The correction. Decompression throughput is normally reported against uncompressed size, where every Parquet variant clears 50 GB/s and looks far faster than a 100 Gbit link needs. Against compressed size — the bytes that actually arrive — Parquet+Zstd manages 24.8 Gbit/s. Compressing better makes this worse, because the same CPU time now covers fewer wire bytes. BtrBlocks reaches 86.2 Gbit/s against a client ceiling of 91, and costs 1.8× less per scan while compressing worse than Parquet+Zstd on real data (5.28× vs 6.05×).

That is a genuine trade this spoke had no vocabulary for: storage cost against scan cost, decided by where the data sits relative to the compute. It belongs next to storage-compute-disaggregation — the axis that made the trade exist in the first place.

And it re-reads an old result. columnar-format-evaluation found the open formats encoded early-2010s assumptions. The assumption that turned out to matter most was not an encoding choice at all; it was that decompression is cheap relative to I/O, which stopped being true when the I/O became 100 Gbit.

A caution the page carries and this section should repeat. The authors built both sides of the comparison. Nothing independent has re-run it, which is the same shape as the vendor benchmarks this spoke discounts — except that the seller here is a university group with no product. Grade it T1 and treat the numbers as unreplicated.

Something arrived that is not an engine (2026-08-31)

slayer (MotleyAI, MIT, 158★, routed 2026-08-31) is the first semantic layer in this spoke and the second source, after jitsu, that sits above every engine the wiki is built around. The two bracket the stack from opposite ends: Jitsu writes events into whichever warehouse you name, SLayer reads out of whichever warehouse is behind it. Neither stores anything. Both treat duckdb, clickhouse, snowflake and their peers as a configuration value.

That is worth recording because the spoke's founding thesis — "not which engine wins, but which engine for which layer of the stack" — assumes the engine choice is the interesting decision. Two sources now model it as substitutable. They do not refute the thesis (both are early, both are one project's design, and neither argues the engine is irrelevant), but they describe a stack in which the engine is the part you swap and the layer around it is the part you keep. Two sources is a thing to watch, not a finding.

The argument SLayer actually makes is narrower and more testable than that. Classical semantic layers bind a metric to its aggregation at definition time — monthly revenue is a defined object. A dashboard user picks from what was defined, so the boundary is invisible. An agent answering an open question crosses it on the first ratio or year-over-year comparison. SLayer's answer moves aggregation to query time (revenue:sum, revenue:avg, revenue:sum / *:count, time_shift(revenue:sum, -1, 'year')) while keeping definitions and row-level security at design time.

And what that trades should be stated, because the source does not. A predefined metric is a guarantee that everyone gets the same number. An expression grammar hands composition back to the caller, and the motivating caller is a language model. The model is still constrained — governed columns, fixed grammar, no free SQL — but consistency stops being structural. Whether that is a good trade is exactly the kind of question this spoke likes and nobody has measured it.

The evidence base is one screenshot. SLayer publishes no benchmark: no accuracy comparison against letting a model write SQL, no translation overhead, no third-party review. The README shows LLM-written SQL beside the equivalent SLayer query for one question on a demo dataset. That is an illustration. Against a spoke whose strongest pages are SIGMOD and VLDB papers with production numbers, this is the thinnest corner in the wiki, and it is thin on a question — should an agent write SQL or call a governed layer? — that is about to matter commercially. New growth edge: any measurement at all on agent-facing analytics interfaces.

One more thing the metadata says. 2,035 commits against 158 stars, sustained over five months. The hub keeps meeting the opposite shape — thousands of stars behind a single push — and it is worth noting that neither number is evidence about the software. Attention and maintenance are separate axes, and this artifact has the second without the first.

Contradictions / tensions

Cross-spoke adjacency

Index — Analytical Databases Wiki

Catalog of every page, grouped by schema.org @type. Spine: synthesis (thesis), log.md (history), this file (catalog). Benchmark/version facts are dated snapshots.

DefinedTerm (concepts)

SoftwareApplication (databases / platforms)

SoftwareSourceCode (sources)

Organization (entities)

ScholarlyArticle / TechArticle / BlogPosting (sources)

Synthesis

Related pages

Links to

Linked from