Log — Analytical Databases Wiki

Append-only history. Each entry starts with ## [YYYY-MM-DD] <op> | <title> where <op> is ingest, query, lint, or split, so grep "^## \[" log.md | tail -5 works.

[2026-06-18] split | analytical-databases-wiki created from _inbox (2 sources)

Spun out at the human's explicit direction. Below the usual ≥3 trigger (2 sources, both ClickHouse-vs-DuckDB comparisons) — recorded as a deliberate user override; founding corpus is thin but the domain (OLAP/analytical databases) is large and clearly growable. Migrated the two parked records (tinybird-clickhouse-vs-duckdb, posthog-duckdb-vs-clickhouse) and paged the three subjects they cover (clickhouse, duckdb, tinybird) plus the analytical-databases umbrella. The two _inbox records were deleted on migration. Domain: analytical/OLAP databases & the data-engineering stack — distinct from ../cloud-wiki (rented infra + pricing), ../platform-ops-wiki (operating prod systems), and ../dev-tooling-wiki (serialization formats). See synthesis.md.

[2026-06-18] ingest | PostHog "DuckDB vs ClickHouse: why we use both" — migrated from _inbox

T2 independent practitioner account. PostHog runs both: clickhouse for product event-analytics (billions of events, streaming ingest + insert-time materialized views) and duckdb for their managed data-warehouse product (per-customer data), wrapped in Duckgres (Postgres-compatible server) on DuckLake + a Postgres catalog, scaled independently of production. Reason for both: self-managed ClickHouse "would not work for data warehousing for all of our customers." New source posthog-duckdb-vs-clickhouse; touched clickhouse/duckdb. The anchor source for synthesis's "use both for different layers" thesis.

[2026-06-18] ingest | Tinybird "ClickHouse vs DuckDB: how many nodes" — migrated from _inbox

T3 vendor blog (Tinybird sells managed ClickHouse; sign-up CTAs — bias recorded). The embedded-vs-distributed split + node-scaling numbers (DuckDB ~100GB single-machine, 1B rows on a 16GB laptop; ClickHouse single-node ~10s vs 4-node ~2-3s on 1TB, 200K→800K inserts/s across 4 nodes). New source tinybird-clickhouse-vs-duckdb; touched clickhouse/duckdb/tinybird. Found via WebSearch at the user's request (original prompt "Tinybird ClickHouse vs DuckDB analytics"). Numbers are indicative vendor figures, not independent.

[2026-06-30] ingest | Snowflake SIGMOD 2016 paper — quality-cycle gap closure

T1 peer-reviewed primary (Dageville et al., Snowflake Computing). Closes the synthesis open-Q "where do the cloud warehouses fit" (partial) — the spoke's first cloud-warehouse source and first T1. Key add: a second axis beyond embedded↔distributed — storage-compute-disaggregation. The paper argues shared-nothing (ClickHouse-style coupled compute+local-disk) is a poor cloud fit (heterogeneous workload, membership-change reshuffles, hard online upgrades) and splits storage (S3/blob) from elastic compute (virtual warehouses, T-shirt sizes, pure/isolated/pay-per-hour) into independent services — the multi-cluster, shared-data architecture. Also captured: immutable table files → MVCC/Snapshot Isolation, time travel (≤90d file retention, SQL AT/BEFORE), UNDROP/zero-copy clone; min-max pruning + local SSD caching instead of indexes; VARIANT/ARRAY/OBJECT semi-structured types; end-to-end encryption + SQL RBAC; pure-SaaS delivery. New pages snowflake-elastic-data-warehouse (source), snowflake, storage-compute-disaggregation, snowflake-computing (Organization entity); touched analytical-databases (added the disaggregated corner) + synthesis (2-D map; reframes the "use both" finding — disaggregation is the other escape from compute-storage coupling). Ingested from the CMU 15-721 mirror (DOI 10.1145/2882903.2903741); 2016 figures are dated snapshots. +4 pages (6→10).

[2026-08-04] ingest | Jitsu (jitsucom)

Routed here by the hub (runner-up: none). T3 — the project's own repository, from a company selling the hosted version; the Segment comparison is competitive positioning by an interested party. 1 new page: jitsu. Updated synthesis (two open questions) and index.

First source on the layer above the engine. Everything the spoke held was about how an analytical database stores or queries; this is about how data gets into one — Ingest/Rotor/Bulker/Sync-controller over Kafka and clickhouse, landing events in clickhouse, BigQuery, snowflake, Redshift, Postgres and S3.

Two things it does to the spine:

1. It widens the engine-vs-managed-service question. tinybird raised it at the serving end; ingestion latency is the same question at the loading end, set by the pipe's batching policy rather than the engine. A fast columnar store behind a twice-daily loader is a slow analytics stack. The engine is now flanked by two commercial layers, both arguing the packaging is the real product, and neither supplying a number — "minutes, not hours" is positioning, not a benchmark.
2. It flattens the spoke's own distinctions. From the pipe's view clickhouse and snowflake differ in loading interface and little else, which the storage-compute-disaggregation axis says should not be true. Filed as a new open question, not a finding: one vendor source, and it is convenient for that vendor.

Also recorded: the billing-shape argument (per-event vs Segment's per-Monthly-Tracked-User; no usage limit self-hosted under MIT), because it is structurally the same move tinybird makes on operational burden.

[2026-08-09] ingest | Parquet, ORC, Arrow, and the paper that dates all three (via research pass)

Coverage edge 4 closed with the specifications plus the benchmark it asked for. parquet (parquet.apache.org format docs), orc (ORC v1 specification), apache-arrow (the columnar format spec) — all T1 first-party — and columnar-format-evaluation, Zeng, Hui, Shen, Pavlo, McKinney and Zhang, VLDB 2023, T1.

The substrate under every engine here had no page. Parquet and ORC turn out to be structural near-twins — row groups against stripes, footers against tails, per-chunk compression, min/max statistics, bloom filters — which is why they can be benchmarked head to head and why the verdict lands on both. Arrow is the one that is different in kind: in-memory, zero-copy across processes and languages, and explicit that it buys read performance with expensive mutation.

The benchmark is the part that changes the spoke's thesis. Zeng et al. find both storage formats were built in the early 2010s for Hadoop and recommend reversing several defaults — dictionary encoding always on, decoding speed over compression ratio, block compression optional, finer skip structures — and report inefficiency on ML workloads and GPU decoding. Two of the authors are Pavlo and McKinney, which is worth stating: this is an argument about the next format, not a neutral audit.

Recorded as a tension in synthesis, because it cuts at storage-compute-disaggregation: open portable files are what let storage and compute separate, and they are also what prevents the format from being improved. Edge 4's successor asks for a next-generation format's design paper.

Entities: 0 created — author and org pages deferred; Pavlo and McKinney will recur, and one paper is not yet a recurrence.

[2026-08-15] ingest | Druid: A Real-time Analytical Data Store (SIGMOD 2014)

Quality-cycle pass. The spoke had the thinnest tiered base in the corpus (n=4 tiered pages), and its top growth edge — the real-time serving family — named exactly the source to fix that. Hunted and found: the Druid whitepaper, fetched from the Apache repo after static.druid.io refused the connection (retried per the transient-error rule, then switched mirror), extracted locally with pypdf.

Created druid-real-time-analytical-data-store (T1: peer-reviewed, by the architects, with production figures) and apache-druid (the engine as a Thing page, in the pattern snowflake/snowflake-elastic-data-warehouse set).

The edge was mis-specified, and the source corrected it. The edge described this family as pre-aggregated, low-latency serving. Druid stores raw events; what buys the latency is a mandatory timestamp column enabling first-level query pruning, plus an in-memory row index that is queryable before columnar conversion. The real axis is freshness — how long after an event arrives you can query it — and synthesis.md's thesis is now 3-D rather than 2-D. The stale description was also corrected inside the "Where do the cloud warehouses fit?" open question, which had been carrying it.

The finding worth keeping is the price of that axis: Druid shipped without joins and argued for the omission rather than treating it as unfinished ("scaling join queries has been… a constant bottleneck of working with distributed databases"). Beside clickhouse and snowflake, which both join, that makes the real-time family a different trade rather than a weaker version of one.

Weaknesses recorded on the page, not smoothed over: the TPC-H comparison is against MySQL/MyISAM only — the authors say plainly they were not confident they could tune a competing column store — so the headline benchmark beats a row store at a column store's job. The production numbers (~550 ms mean, 99% under 10 s, ~50B rows against 10.5 TB of RAM) are the useful half, and they describe one company's cluster in early 2014. A new open question records what the paper cannot answer: Druid has since added SQL and ClickHouse has moved toward real-time ingestion, so whether freshness still costs the query surface is unsourced here.

[2026-08-16] ingest | Pinot: Realtime OLAP for 530 Million Users (SIGMOD 2018) — via research pass

Hunted against growth edge 1's successor (the real-time family beside a general engine), which the hub had promoted to ## Most wanted #20 on 2026-08-15.

pinot-realtime-olap written (T1, SIGMOD'18, 11 LinkedIn authors; PDF extracted locally with pypdf rather than trusting a fetch summary — the two disagreed on the production figures and the paper won). Thing pages apache-pinot and linkedin (Organization) created; apache-druid and druid-real-time-analytical-data-store gain the other side of a comparison they could not make.

What it closes. The family had one design in it and now has two, from two teams, four years apart — and the second paper benchmarks against the first: nine hosts, Xeon E5-2680 v3, 64 GiB, 2.8 TB NVMe, production data and sampled production queries, realtime ingestion disabled on both sides. Findings kept: Druid ≈ Pinot with no indexes; Druid's index-every-dimension default costs 1.2 TB against Pinot's 300 GB on one dataset; Pinot credits physical row ordering over index structure on its highest-throughput case and star-tree pre-aggregation on the ad-hoc case.

What it does not close, recorded on the page and the edge. The comparison is run by Pinot's authors, the mirror of Druid's paper declining to tune a rival, and the results are plots rather than tables, so this spoke quotes directions and ratios rather than latencies. StarRocks was hunted and refused: no paper exists, and every StarRocks-vs-ClickHouse benchmark in reach is published by StarRocks/CelerData or by Tinybird, which sells managed ClickHouse. The edge now asks for a real-time engine measured against clickhouse by a party selling neither.

synthesis gains Two designs in one family, and one of them tuned the other.

[2026-08-22] ingest | BtrBlocks: Efficient Columnar Compression for Data Lakes — via research pass

btrblocks (Kuschewski, Sauerwein, Alhomssi & Leis, TUM/FAU, SIGMOD 2023, T1), read from the authors' PDF. Closes coverage edge 4's successor — the next-generation format columnar-format-evaluation pointed at and the spoke did not hold.

The useful part is a metric, not a format. Decompression throughput is normally quoted against uncompressed size, where every parquet variant clears 50 GB/s and decompression looks free next to a 100 Gbit link. Against compressed size — the bytes that actually cross the network — Parquet+Zstd manages 24.8 Gbit/s and the scan is CPU-bound. BtrBlocks reaches 86.2 Gbit/s against a 91 Gbit/s client ceiling.

And it loses the headline number. On the Public BI Benchmark BtrBlocks compresses 5.28× against Parquet+Zstd's 6.05×, and still costs 1.8× less per S3 scan. Storage ratio and scan cost pull in opposite directions once the data is remote — a trade this spoke had no vocabulary for. Written into synthesis beside storage-compute-disaggregation.

Design notes on the page: seven encodings plus a new Pseudodecimal scheme for doubles, cascading, sampling-based scheme selection (~1% of a 64k block, in scattered chunks), and metadata deliberately left out of the data file so it can be pruned before a high-latency request.

Caveats recorded, not buried. The authors built both sides of the comparison and nobody has re-run it; one instance type, one cloud, 2023 prices; no query engine in the loop. Successor written as edge 8 — Lance, Vortex and Nimble are still unsourced and nothing compares any two of them.

[2026-08-30] research | Real-time Data Infrastructure at Uber (research pass, edge 1)

Hunted growth edge 1 — "a real-time engine measured against clickhouse by someone selling neither." Came back with the operator half and not the ClickHouse half, so the edge stays open and a successor was written.

realtime-data-infrastructure-uber (Fu & Soman, SIGMOD 2021, arXiv 2104.00087, T1, read from the PDF with pypdf after the abstract page proved lossy). What it supplies that nothing here had: an operating envelope rather than a benchmark — trillions of messages and petabytes per day, Pinot growing from dozens of GB to several hundred TB and from a few hundred QPS to tens of thousands over two years, under a p99 under one second requirement driven by a user-facing page load.

The numbers it does publish are against Elasticsearch: 4× the memory, 8× the disk, 2–4× the query latency on the same ingested data. Against Druid there are none — and the sentence that carries the decision cites "recent studies done outside of Uber," one of which is a Medium post. That is the spoke's own recurring finding turning up inside a peer-reviewed paper by the world's largest Pinot operator, and it narrows the evidence base behind the Pinot-over-Druid claim this wiki has been repeating. Written as successor edge 1a.

ClickHouse is mentioned once, in related work, unevaluated. Edge 1 not closed.

[2026-08-31] ingest | SLayer — the spoke's first semantic layer

Routed from the hub (Telegram). https://github.com/MotleyAI/slayer — MIT, Python, 158★ / 22 forks, 2,035 commits, created 2026-03-17, pushed 2026-08-30. T1 for the artifact's own facts; it publishes no benchmark. Runner-up spoke: agentic-tooling-wiki (the MCP server and the search-inspect-query flow), recorded as cross-spoke context rather than paged there.

Pages: new slayer, semantic-layer (concept), motleyai (entity). Updated analytical-databases (a new section on the layer above the engines), synthesis.md, index.md.

Why it is not a category error. Everything here so far is a place data lives or a service wrapping one. SLayer stores nothing: it is read-only, compiles an expression into SQL, and lets duckdb/clickhouse/snowflake execute it. It is the read counterpart to jitsu, which the index already called the first source on the layer above the engine — Jitsu writes into any warehouse, SLayer reads out of any warehouse.

The claim worth keeping. Classical semantic layers bind metric to aggregation at definition time; an agent crosses that boundary on the first ratio or time shift. SLayer defines the column once and composes the aggregation at query time. What it trades — and the README does not say this — is that a predefined metric guarantees everyone gets the same number, while an expression grammar returns composition to a caller that is a language model.

Evidence: one screenshot. No benchmark, no accuracy comparison against text-to-SQL, no overhead numbers. Written as a new growth edge, since the question is commercially live and the spoke's other pages are SIGMOD/VLDB papers with production figures.