Synthesis — Analytical Databases

The evolving thesis. Spun out of the hub _inbox analytical-databases cluster on 2026-06-18 at the human’s direction — a deliberate early spin-out at two sources (below the usual ≥3), both ClickHouse-vs-DuckDB comparisons. So the founding corpus is narrow: one axis (embedded vs distributed OLAP), seen once through a vendor lens and once through an independent practitioner lens. The domain is much larger than that, and the thesis below is a starting frame to test as warehouses, real-time platforms, and more engines arrive.

Current thesis

Analytical (OLAP) databases sit on an embedded↔distributed spectrum, and the right answer is usually a position on it — often more than one position at once.

• duckdb anchors the embedded / in-process end: a library inside your app, single machine, zero ops. Strong for notebooks, dev, edge, and datasets that fit one box (≈100 GB; ~1B rows on a 16 GB laptop). The “SQLite for analytics.”
• clickhouse anchors the distributed columnar end: shared-nothing MergeTree across nodes, for high-ingest (100K+ rows/s), high-concurrency, RAM-exceeding, fault-tolerant production analytics.
• tinybird is the managed-service move on top of ClickHouse — abstract shards/replication/ routing and add an API serving layer, trading control for less ops.
• snowflake opens a second axis the embedded↔distributed line missed: storage-compute-disaggregation. Its SIGMOD 2016 paper argues the shared-nothing design ClickHouse uses (compute tied to local disk) is a poor cloud fit, and splits storage (object store) from compute (elastic “virtual warehouses”) into independently scalable services. So the map is now 2-D: where on embedded↔distributed and coupled vs disaggregated. Both clickhouse and snowflake are “distributed,” but classic ClickHouse is coupled while Snowflake is disaggregated — that’s the sharper distinction than “single box vs many boxes.”

Added 2026-06-30: the warehouse end is now sourced (Snowflake, T1). It reframes the founding “use both” finding: PostHog reached for embedded DuckDB because a coupled engine didn’t fit per-tenant warehousing — disaggregation is the other escape hatch from the same coupling problem.

The sharpest finding so far: the “vs” framing is partly false. The independent PostHog account is the load-bearing source — at production scale they run both, for different layers: ClickHouse for their own product event-analytics, and DuckDB (wrapped in Duckgres on DuckLake + a Postgres catalog) for a per-customer data-warehouse product that had to scale independently, because self-managed ClickHouse “would not work for data warehousing for all of our customers.” So the practitioner question isn’t “which wins” but “which engine for which layer of the stack” — embedded per-tenant warehousing vs centralized high-scale event analytics.

Open questions

• Is “use both” the general pattern or a PostHog-specific architecture? Needs more practitioner accounts before generalizing.
• Where do the cloud warehouses fit? Partly answered (2026-06-30): snowflake is now sourced from its SIGMOD 2016 paper (T1) — they fit via a new storage-compute-disaggregation axis (elastic compute over shared object storage), not just a point on embedded↔distributed. Still open: BigQuery (Dremel/Colossus) and Databricks (lakehouse) are the same disaggregated family but unsourced; real-time engines (Pinot, Druid, StarRocks) are a different shape (pre-aggregated, low-latency serving) and entirely unsourced. Next most-valuable adds.
• How much of the gap is the engine vs the managed service? tinybird reframes “ClickHouse is hard to operate” as a serving problem; the engine-vs-ops distinction needs its own sources. Widened 2026-08-04 (jitsu). The same question exists at the other end of the pipeline: ingestion latency is set by the loader’s batching policy, not the engine, and a fast columnar store behind a twice-daily loader is a slow analytics stack. So the spoke now has the engine flanked by two commercial layers — serving (tinybird) and ingestion (jitsu) — each arguing that the packaging around the engine is the real product. Neither supplies a measurement. That is starting to look like the spoke’s actual subject rather than a distraction from it.
• Do the engine distinctions survive contact with the pipe? New 2026-08-04. jitsu treats clickhouse, snowflake, BigQuery, Redshift, Postgres and S3 as interchangeable destinations differing mainly in loading interface. The storage-compute-disaggregation axis says these are architecturally distinct in ways that should matter; at the ingestion layer, apparently they do not. Whether that is a real finding about where the differences live, or just what a pipe vendor needs to be true, is unresolved on one source.
• The OLTP boundary. This spoke is analytical/OLAP; where HTAP / Postgres-as-analytics (DuckLake’s Postgres catalog hints at it) blurs the line is an open edge.

Growth edges

Ranked; each names the kind of source that would close it (see ../QUALITY.md → Growth edges).

1. The real-time serving family. Pinot, Druid and StarRocks are a different shape from the embedded↔distributed axis this spoke was founded on (pre-aggregated, low-latency serving) and are entirely unsourced. — needs: a T1 architecture paper or first-party design doc for one of them.
2. The rest of the disaggregated family. Snowflake is sourced from its SIGMOD paper; BigQuery (Dremel/Colossus) and Databricks (lakehouse) are the same family and unsourced. — needs: the Dremel or lakehouse paper (T1).
3. Engine versus packaging, measured. tinybird (serving) and jitsu (ingestion) both argue the packaging around the engine is the real product, and neither supplies a number. — needs: any measurement separating engine performance from the managed layer.

Coverage edges (added 2026-08-08, at the curator’s request for a wider backlog). These widen what the spoke covers instead of answering an open question above; one ordinary solid source closes any.

4. The file formats everything rests on. CLOSED 2026-08-09 (research pass) — parquet, orc and apache-arrow from their own specifications, plus columnar-format-evaluation (Zeng et al., VLDB 2023) as the benchmark, all T1. Successor: what replaces them. The paper says the standard formats encode early-2010s assumptions; several projects (BtrBlocks, Lance, Vortex, Nimble) claim to be the answer and the spoke holds none of them. — needs: a next-generation format’s design paper, T1/T2.
5. Open table formats. Iceberg, Delta Lake and Hudi are where the lakehouse claim is actually made, and storage-compute-disaggregation has no counterpart page for the metadata layer that makes the shared storage writable. — needs: a specification or design paper.
6. A query engine that owns no storage. Trino and Presto separate execution from the store, the clean case for the disaggregation axis. — needs: the Presto SIGMOD paper or Trino’s own docs.
7. What people put in these engines. dbt and dimensional modelling (Kimball) shape more analytical workloads than engine choice does, and the spoke covers only engines. — needs: dbt’s documentation or a modelling text.

Contradictions / tensions

• Portability freezes the format (2026-08-09). storage-compute-disaggregation works because any engine can read the same open files, and columnar-format-evaluation shows the cost of that guarantee: parquet and orc carry design decisions from the early-2010s Hadoop world — compression ratio favoured over decoding speed, dictionary encoding optional, coarse skip structures — and the paper measures them failing on ML workloads and GPU decoding, neither of which existed when the formats were designed. An engine can rewrite its execution layer whenever it likes; it cannot change the format without leaving the ecosystem that made shared storage valuable. So the same property this spoke credits for the lakehouse is the property preventing the substrate from improving. Not a contradiction between two sources — both agree on the facts — but a real tension inside the spoke’s own thesis. No direct fact conflict between the two founding sources — but a framing tension worth flagging: the vendor T3 frames ClickHouse as the production answer (and Tinybird as the way to run it), while the independent T2 shows a real shop reaching for DuckDB precisely because ClickHouse didn’t fit a major workload. Weight the independent account more; the vendor numbers are indicative, not neutral.

Cross-spoke adjacency

• cloud-wiki — where you rent the machines/managed services these run on (pricing/IaaS); this spoke is the database as subject.
• platform-ops-wiki — the SRE/observability practice of operating prod stores (ClickHouse cluster ops, ZooKeeper/replica management); here it’s the engine, there it’s running it.
• dev-tooling-wiki — serialization/wire formats (Protobuf/yaff) move data between systems; an OLAP database stores and queries it — adjacent data layers, different jobs.

Index — Analytical Databases Wiki

Catalog of every page, grouped by schema.org @type. Spine: synthesis (thesis), log.md (history), this file (catalog). Benchmark/version facts are dated snapshots.

DefinedTerm (concepts)

• analytical-databases — umbrella: OLAP/columnar query engines across the embedded↔distributed spectrum; the axes this spoke tracks · domain

• storage-compute-disaggregation — the coupled-vs-disaggregated axis (shared-nothing vs object-store + elastic compute); orthogonal to embedded↔distributed

• parquet — the on-disk columnar format: row groups → column chunks → pages, footer-last so a file can be written in one pass, metadata deliberately separable from data. The substrate the lakehouse claim assumes · source · T1 · parquet.apache.org

• orc — Hive’s type-aware columnar format: ~200 MB stripes that never split a row, min/max statistics every 10,000 rows plus bloom filters, an uncompressed ≤256-byte postscript to bootstrap the reader. Near-twin of Parquet in structure, different lineage · source · T1 · orc.apache.org

• apache-arrow — the in-memory columnar format: validity bitmap + buffers, aligned for SIMD, “relocatable without pointer swizzling” so sharing is zero-copy across processes and languages. Buys read performance at the cost of expensive mutation · source · T1 · arrow.apache.org

SoftwareApplication (databases / platforms)

• duckdb — embedded, in-process analytical (OLAP) database; single-machine, zero-ops; the “SQLite for analytics”
• clickhouse — distributed columnar OLAP database (MergeTree); horizontal scale for high-ingest, high-concurrency analytics
• snowflake — cloud data warehouse; storage-compute disaggregated, elastic “virtual warehouses,” pay-per-use SaaS
• tinybird — managed ClickHouse + an API serving layer (hides shards/replication/routing) for real-time analytics

SoftwareSourceCode (sources)

• jitsu — jitsucom, MIT, 4.9k★: open-source event data platform (“Segment alternative”) — Ingest/Rotor/Bulker/Sync-controller/Console over Kafka + ClickHouse; lands events in ClickHouse, BigQuery, Snowflake, Redshift, Postgres, S3. The spoke’s first source on the layer above the engine; treats the warehouses as interchangeable endpoints · source · T3 · github.com

Organization (entities)

• snowflake-computing — the company behind snowflake (founded 2012; AWS GA June 2015)

ScholarlyArticle / TechArticle / BlogPosting (sources)

• columnar-format-evaluation — Zeng, Hui, Shen, Pavlo, McKinney & Zhang, VLDB 2023 (T1): Parquet vs ORC stress-tested — both “developed over a decade ago, in the early 2010s, for the Hadoop ecosystem.” Recommends dictionary encoding by default, decoding speed over compression ratio, optional block compression, finer-grained skip structures; finds them inefficient for ML workloads and GPU decoding · source · T1 · arxiv.org
• snowflake-elastic-data-warehouse — Snowflake’s SIGMOD 2016 paper: the multi-cluster shared-data architecture; storage/compute separation · source · T1 · cs.cmu.edu (mirror)
• posthog-duckdb-vs-clickhouse — PostHog eng blog: “why we use both” — ClickHouse for product analytics, DuckDB for the customer warehouse · source · T2 · posthog.com
• tinybird-clickhouse-vs-duckdb — Tinybird: ClickHouse vs DuckDB node-scaling comparison (vendor) · source · T3 · tinybird.co

Synthesis

• synthesis — the evolving thesis (open questions + flagged contradictions)