Apache Pinot

A distributed, column-oriented real-time OLAP datastore built at LinkedIn and later an Apache project. Documented here from its SIGMOD 2018 paper; everything below describes the system as that paper does, and the years of development since are not sourced.

Pinot is the spoke's second engine from the real-time serving family, after apache-druid, and the pair is the point: they share a lineage, a target workload and most of their architecture, and they disagree about the details that turn out to matter.

What it is for

Serving analytical queries to the site's own users rather than to analysts. LinkedIn's examples are "who viewed my profile," content analytics for publishers, and impression discounting on the news feed — every feed view issuing several queries and emitting events that must be queryable almost immediately. Query volume therefore scales with the user base, which is where the paper's title comes from and why 50,000 queries per second is a routine figure rather than a peak.

Architecture, in the paper's terms

Segments — immutable columnar units, the unit of assignment and replication.
Servers hold segments and execute the query fragments; brokers scatter a query across the servers holding relevant segments and gather results; controllers own cluster state and segment assignment.
Two table types, one logical table — a realtime table ingesting from Kafka for freshness and an offline table loaded from Hadoop for history, with segments ageing over from one to the other.
Multitenancy by token bucket, so tenants sharing hardware queue rather than starve each other.

The three levers it pulls

Ranked by what the paper's own measurements credit:

Star-tree index — hierarchical pre-aggregation with a tunable threshold, trading storage for a bounded number of records scanned. The biggest gain on the ad-hoc/anomaly-detection workload.
Physical row ordering — sorting rows on the identifier that queries filter by. On the highest-throughput dataset this, not the inverted index, is what separates Pinot from apache-druid; both use roaring bitmaps for their inverted indexes.
Partition-aware broker routing — cutting the fan-out so latency stays flat as query rate rises.

Against Druid

Same family, different defaults, and the difference shows up as size: Druid builds an inverted index on every dimension column, Pinot indexes selectively, so on one production dataset Druid held 1.2 TB where Pinot held 300 GB. Without indexes the two perform comparably; Pinot's advantage is what it adds on top. The comparison comes from Pinot's own authors — see pinot-realtime-olap for why that cuts both ways.

As operated somewhere other than LinkedIn

Every number above comes from Pinot's own authors at LinkedIn. The one independent-of-LinkedIn account of Pinot in production is realtime-data-infrastructure-uber (Fu & Soman, SIGMOD 2021), and it is an operator's report rather than a vendor's: Pinot at Uber grew from dozens of GB to several hundred TB and from a few hundred to tens of thousands of QPS in two years, under a requirement of p99 query latency under one second because the queries run per page load. That paper measures Pinot against Elasticsearch — 4× the memory, 8× the disk, 2–4× the latency — and against apache-druid not at all, citing a Medium post for the choice.

realtime-data-infrastructure-uber · pinot-realtime-olap · apache-druid · clickhouse · analytical-databases · linkedin · synthesis