Real-time Data Infrastructure at Uber (SIGMOD 2021)

Yupeng Fu and Chinmay Soman, Uber. SIGMOD '21, arXiv 2104.00087, 14 pages, read from the PDF with pypdf. T1 — a peer-reviewed operator account of a production stack, not a vendor benchmark.

The spoke went looking for Pinot or a real-time engine measured against clickhouse by somebody selling neither. This is the operator half of that ask, and what it actually delivers is different from what was hoped for — and more interesting.

The scale, which is the part nobody else supplies

• Trillions of messages and petabytes per day across all regions, as of October 2020, through one of the largest Kafka deployments in industry.
• Pinot at Uber went from dozens of GB to several hundred TB in two years, and from a few hundred QPS to tens of thousands of QPS over the same period.
• The binding requirement is p99 query latency under one second, because the queries run per page load in a user-facing tool (the UberEats Restaurant Manager), several per load.

Every other real-time source this spoke holds reports a benchmark. This one reports an operating envelope, which is the number practitioners actually need when choosing an engine.

The selection, and what it was decided on

Uber chose Pinot in 2018 against a field of two: Elasticsearch and Druid. Only one of those comparisons carries numbers.

Against Elasticsearch, on the same data ingested into both: Elasticsearch used 4× the memory and 8× the disk, and ran queries 2–4× slower on a mix of filters, aggregations and group-by/order-by. Those are Uber's own measurements and they are the first hard numbers this spoke holds for a real-time engine against a search engine, which is the comparison a lot of teams genuinely face.

Against Druid, there are no numbers at all. The paper says Pinot is architecturally similar but carries bit-compressed forward indices for a smaller footprint and specialised indices — Star-Tree, sorted, range — "which could result in order of magnitude difference of query latency." Then it cites the evidence: "Recent studies done outside of Uber also confirmed this performance benefit of Pinot over Druid."

The citation worth stopping on

Reference [47], one of the two studies backing that sentence, is a Medium post — Roman Leventov's "Comparison of the Open Source OLAP Systems for Big Data: ClickHouse, Druid, and Pinot."

So the largest Pinot operator in the world, publishing at SIGMOD, supports its Pinot-over-Druid choice by pointing at a blog. Not because the team is careless — they clearly ran their own Elasticsearch comparison and published the ratios — but because at the moment of the decision there was nothing better to point at, and there still is not. This is the corpus's recurring finding arriving from an unusually credible direction: the independent measurement is missing even where the stakes are petabytes and the author is a peer-reviewed practitioner.

It also means the Pinot-beats-Druid claim in this spoke, in pinot-realtime-olap and now here, traces to the same small pool of sources rather than to two independent confirmations.

What it does not answer

ClickHouse appears once, in related work, listed beside Druid as a commonly adopted open-source real-time OLAP system. It is not evaluated, not benchmarked, and not part of the 2018 selection — Uber's field was Elasticsearch and Druid. So the edge this hunt was aimed at, a real-time engine measured against ClickHouse by a disinterested party, is still open. What closed is the neighbouring question of what such an engine looks like under production load, and against Elasticsearch.

Related

apache-pinot · apache-druid · clickhouse · pinot-realtime-olap · druid-real-time-analytical-data-store · analytical-databases · synthesis