Log — Analytical Databases Wiki

Append-only history. Each entry starts with ## [YYYY-MM-DD] <op> | <title> where <op> is ingest, query, lint, or split, so grep "^## \[" log.md | tail -5 works.

[2026-06-18] split | analytical-databases-wiki created from _inbox (2 sources)

Spun out at the human’s explicit direction. Below the usual ≥3 trigger (2 sources, both ClickHouse-vs-DuckDB comparisons) — recorded as a deliberate user override; founding corpus is thin but the domain (OLAP/analytical databases) is large and clearly growable. Migrated the two parked records (tinybird-clickhouse-vs-duckdb, posthog-duckdb-vs-clickhouse) and paged the three subjects they cover (clickhouse, duckdb, tinybird) plus the analytical-databases umbrella. The two _inbox records were deleted on migration. Domain: analytical/OLAP databases & the data-engineering stack — distinct from cloud-wiki (rented infra + pricing), platform-ops-wiki (operating prod systems), and dev-tooling-wiki (serialization formats). See synthesis.md.

[2026-06-18] ingest | PostHog “DuckDB vs ClickHouse: why we use both” — migrated from _inbox

T2 independent practitioner account. PostHog runs both: clickhouse for product event-analytics (billions of events, streaming ingest + insert-time materialized views) and duckdb for their managed data-warehouse product (per-customer data), wrapped in Duckgres (Postgres-compatible server) on DuckLake + a Postgres catalog, scaled independently of production. Reason for both: self-managed ClickHouse “would not work for data warehousing for all of our customers.” New source posthog-duckdb-vs-clickhouse; touched clickhouse/duckdb. The anchor source for synthesis‘s “use both for different layers” thesis.

[2026-06-18] ingest | Tinybird “ClickHouse vs DuckDB: how many nodes” — migrated from _inbox

T3 vendor blog (Tinybird sells managed ClickHouse; sign-up CTAs — bias recorded). The embedded-vs-distributed split + node-scaling numbers (DuckDB ~100GB single-machine, 1B rows on a 16GB laptop; ClickHouse single-node ~10s vs 4-node ~2-3s on 1TB, ~200K→~800K inserts/s across 4 nodes). New source tinybird-clickhouse-vs-duckdb; touched clickhouse/duckdb/tinybird. Found via WebSearch at the user’s request (original prompt “Tinybird ClickHouse vs DuckDB analytics”). Numbers are indicative vendor figures, not independent.

[2026-06-30] ingest | Snowflake SIGMOD 2016 paper — quality-cycle gap closure

T1 peer-reviewed primary (Dageville et al., Snowflake Computing). Closes the synthesis open-Q “where do the cloud warehouses fit” (partial) — the spoke’s first cloud-warehouse source and first T1. Key add: a second axis beyond embedded↔distributed — storage-compute-disaggregation. The paper argues shared-nothing (ClickHouse-style coupled compute+local-disk) is a poor cloud fit (heterogeneous workload, membership-change reshuffles, hard online upgrades) and splits storage (S3/blob) from elastic compute (virtual warehouses, T-shirt sizes, pure/isolated/pay-per-hour) into independent services — the multi-cluster, shared-data architecture. Also captured: immutable table files → MVCC/Snapshot Isolation, time travel (≤90d file retention, SQL AT/BEFORE), UNDROP/zero-copy clone; min-max pruning + local SSD caching instead of indexes; VARIANT/ARRAY/OBJECT semi-structured types; end-to-end encryption + SQL RBAC; pure-SaaS delivery. New pages snowflake-elastic-data-warehouse (source), snowflake, storage-compute-disaggregation, snowflake-computing (Organization entity); touched analytical-databases (added the disaggregated corner) + synthesis (2-D map; reframes the “use both” finding — disaggregation is the other escape from compute-storage coupling). Ingested from the CMU 15-721 mirror (DOI 10.1145/2882903.2903741); 2016 figures are dated snapshots. +4 pages (6→10).

[2026-08-04] ingest | Jitsu (jitsucom)

Routed here by the hub (runner-up: none). T3 — the project’s own repository, from a company selling the hosted version; the Segment comparison is competitive positioning by an interested party. 1 new page: jitsu. Updated synthesis (two open questions) and index.

First source on the layer above the engine. Everything the spoke held was about how an analytical database stores or queries; this is about how data gets into one — Ingest/Rotor/Bulker/Sync-controller over Kafka and clickhouse, landing events in clickhouse, BigQuery, snowflake, Redshift, Postgres and S3.

Two things it does to the spine:

It widens the engine-vs-managed-service question. tinybird raised it at the serving end; ingestion latency is the same question at the loading end, set by the pipe’s batching policy rather than the engine. A fast columnar store behind a twice-daily loader is a slow analytics stack. The engine is now flanked by two commercial layers, both arguing the packaging is the real product, and neither supplying a number — “minutes, not hours” is positioning, not a benchmark.
It flattens the spoke’s own distinctions. From the pipe’s view clickhouse and snowflake differ in loading interface and little else, which the storage-compute-disaggregation axis says should not be true. Filed as a new open question, not a finding: one vendor source, and it is convenient for that vendor.

Also recorded: the billing-shape argument (per-event vs Segment’s per-Monthly-Tracked-User; no usage limit self-hosted under MIT), because it is structurally the same move tinybird makes on operational burden.

[2026-08-09] ingest | Parquet, ORC, Arrow, and the paper that dates all three (via research pass)

Coverage edge 4 closed with the specifications plus the benchmark it asked for. parquet (parquet.apache.org format docs), orc (ORC v1 specification), apache-arrow (the columnar format spec) — all T1 first-party — and columnar-format-evaluation, Zeng, Hui, Shen, Pavlo, McKinney and Zhang, VLDB 2023, T1.

The substrate under every engine here had no page. Parquet and ORC turn out to be structural near-twins — row groups against stripes, footers against tails, per-chunk compression, min/max statistics, bloom filters — which is why they can be benchmarked head to head and why the verdict lands on both. Arrow is the one that is different in kind: in-memory, zero-copy across processes and languages, and explicit that it buys read performance with expensive mutation.

The benchmark is the part that changes the spoke’s thesis. Zeng et al. find both storage formats were built in the early 2010s for Hadoop and recommend reversing several defaults — dictionary encoding always on, decoding speed over compression ratio, block compression optional, finer skip structures — and report inefficiency on ML workloads and GPU decoding. Two of the authors are Pavlo and McKinney, which is worth stating: this is an argument about the next format, not a neutral audit.

Recorded as a tension in synthesis, because it cuts at storage-compute-disaggregation: open portable files are what let storage and compute separate, and they are also what prevents the format from being improved. Edge 4’s successor asks for a next-generation format’s design paper.

Entities: 0 created — author and org pages deferred; Pavlo and McKinney will recur, and one paper is not yet a recurrence.