Apache Druid
A distributed, column-oriented, real-time analytical data store — built at Metamarkets, open-sourced, now an Apache project. Documented here from its SIGMOD 2014 paper; the pages below describe the system as that paper does, and a decade of subsequent development is not sourced.
Druid is the spoke's first engine from the real-time serving family, and it belongs to a different lineage than the two the spoke was founded on. duckdb and clickhouse answer how fast can I query what I have; Druid answers how fast can I query what just happened, for hundreds of people at once.
What it is optimized for
- Ingest-to-queryable in the hundreds of milliseconds. Events land in an in-memory index that is immediately queryable, before any columnar conversion has happened.
- Sub-second aggregate latency under concurrency — ~550 ms average, 90% under 1 s, on a production cluster serving several hundred users.
- Arbitrary drill-down over a fixed set of dimensions, via nested boolean filter sets and an inverted index, rather than arbitrary SQL.
What it gives up
- Joins. Not implemented as of the paper, and argued for rather than apologized for — the team judged distributed join scaling a worse problem than the missing feature.
- A general query language. Queries are JSON POSTs against its own API, not SQL. (SQL arrived later; this spoke has no source for it.)
- Schema freedom. Every row must carry a timestamp. That requirement is the source of the latency advantage, not an incidental design choice.
Where it sits on this spoke's axes
On embedded↔distributed it is firmly distributed, beside clickhouse. On storage-compute-disaggregation it is an interesting hybrid: segments live in "deep storage" (S3 or HDFS) exactly as the disaggregated warehouses hold data, but queries are served from historical nodes that have already pulled the segments onto local disk and into page cache. Deep storage is the durability and redistribution layer, not the query path. So Druid disaggregates durability from compute while keeping serving coupled — which is why it needs a cluster sized to hold the hot data in RAM (10.5 TB for ~10 TB of segments) where snowflake does not.
The third axis this engine forces open is freshness: how long after an event arrives can you query it. snowflake and the warehouse family answer in ETL batches; Druid answers in milliseconds, and sacrifices the join to do it.
Related
druid-real-time-analytical-data-store · analytical-databases · clickhouse · duckdb · snowflake · storage-compute-disaggregation · tinybird · synthesis