Pinot: Realtime OLAP for 530 Million Users (SIGMOD 2018)

Im, Gopalakrishna, Subramaniam, Shrivastava, Tumbde, Jiang, Dai, Lee, Pawar, Li and Aringunram — eleven authors, all LinkedIn — SIGMOD'18, Houston, 10–15 June 2018. The system paper for apache-pinot, and the second real-time engine this spoke holds after druid-real-time-analytical-data-store. Read from the PDF (12 pages), extracted locally.

Ingested via the hub research pass against the real-time family beside a general engine edge.

The problem it states

The framing is not "OLAP is slow." It is that three existing families each fail one leg of the same requirement: relational databases and key-value stores scale to point lookups for many users but fall apart at "high ingest rates, high query rates at low latency for analytical queries"; online analytical databases assume bulk loads and are not built for nonstop web operation; offline analytical systems have throughput but neither low latency nor tens of thousands of queries per second.

The user-facing consequence is what makes it different from a warehouse paper: the queries are for LinkedIn's own members — "who viewed my profile", content analytics, feed impression discounting — so query volume scales with users, not with analysts.

Production scale at LinkedIn (§5)

• 3,000+ hosts, geographically distributed
• 1,500+ tables, over one million segments
• ~30 TiB compressed, excluding replication
• >50,000 queries per second across all data centres
• Multitenancy by token bucket: several thousand tables across 50+ tenants, each query deducting tokens proportional to its execution time, so a misbehaving tenant queues rather than starves its neighbours

The table counts in §4.5 and §5 do not agree ("several thousand" against "over 1500") — read them as different scopes rather than one number.

The head-to-head with Druid (§6) — what this spoke was missing

This is the part the edge asked for. Pinot's team benchmarked Pinot against Druid on production data and production queries, with the hardware stated:

• Nine hosts for query processing, single-socket Xeon E5-2680 v3 @ 2.50 GHz, 64 GiB RAM, 2.8 TB NVMe, RHEL 6.9; three brokers each side.
• Datasets and queries pulled from LinkedIn production, whole datasets, queries sampled in the tens of thousands.
• Realtime ingestion disabled on both systems — the authors say a repeatable shared realtime environment was too complex to build, so the comparison is of the query path only.

What it found, across three scenarios:

• Anomaly detection / ad-hoc reporting (16 GB Pinot, 13.8 GB Druid): Druid's latency "quickly becomes too high for interactive purposes" as query rate climbs, and so does Pinot with no indexes. Run sequentially rather than under load, "Druid has comparable performance with Pinot, when there are no indexes in Pinot." Inverted indexes roughly double Pinot's scalability here; the star-tree index is the largest gain.
• Share analytics (300 GB Pinot, 1.2 TB Druid): the size gap is itself the finding — Druid builds an inverted index on every dimension column whether or not it is filtered on. Pinot's advantage here is credited mostly to physical row ordering (sorted on the shared item identifier), not to indexing.
• Impression discounting: Druid does "significantly better" here than on the other datasets but scales worse; Pinot's gain comes from partition-aware broker routing, which flattens the latency curve as query rate rises.

Read the comparison for what it is. It is Pinot's authors configuring and tuning a rival, which is the mirror image of the weakness recorded on druid-real-time-analytical-data-store — that paper benchmarked only against MySQL/MyISAM because its authors did not trust their own tuning of a rival column store. Both papers are honest about the limitation and neither escapes it. Results are also presented as plots without a table of figures, so the pages here quote directions and ratios the text states, not latencies read off a chart.

The mechanisms worth carrying

• Star-tree index — a hierarchical pre-aggregated structure with a configurable threshold, so the engine trades storage for a bounded number of records scanned. The paper measures the ratio of pre-aggregated to raw records scanned and finds most queries touch substantially fewer.
• Sorted physical ordering beating bitmap inverted indexes on the highest-throughput dataset — layout over index structure.
• Partition-aware routing at the broker, cutting fan-out overhead at high query rates.
• Self-service operations: schemas change on the fly (new columns backfilled with defaults, live in minutes), query logs are parsed to add inverted indexes automatically, and table configs live in source control synced through the REST API for audit and code review.

Related

apache-pinot · apache-druid · druid-real-time-analytical-data-store · clickhouse · analytical-databases · linkedin · synthesis