The Snowflake Elastic Data Warehouse (SIGMOD 2016)

The paper Dageville et al. (Snowflake Computing) presented at SIGMOD 2016 — the canonical primary account of the cloud data warehouse, and the spoke’s first source for the warehouse end of the OLAP spectrum. T1: peer-reviewed, by the system’s own architects, describing a system generally available since June 2015 and by then “several million queries per day over multiple petabytes.”

The argument

Traditional warehouses were built for small, fixed, well-behaved clusters and deep ETL pipelines — a poor fit for the cloud, where node failures and resizing are the norm, not the exception, and a growing share of data arrives schema-less and semi-structured (logs, web/mobile, IoT). Big-Data platforms (Hadoop, Spark) handle the scale but “lack much of the efficiency and feature set of established data warehousing technology” and take real engineering effort to run. Snowflake’s bet: build a warehouse from scratch for the cloud — not on Hadoop or Postgres.

The key design choice: separate storage and compute

The paper’s load-bearing claim (its Section 2) is that the shared-nothing architecture that dominates high-performance warehousing — every node owns its local disk, tables horizontally partitioned across nodes — has one fatal cloud drawback: it tightly couples compute and storage. That coupling hurts in exactly the situations the cloud makes routine:

• Heterogeneous workload — bulk load wants high I/O + light compute; complex queries want the reverse. One fixed hardware config is a compromise with low average utilization.
• Membership changes — resizing or node failure forces large data reshuffles, and the same nodes do both the shuffling and the query work, so elasticity and availability suffer.
• Online upgrade — tight coupling and assumed homogeneity make zero-downtime upgrades very hard.

So Snowflake splits the system into two loosely coupled, independently scalable services: a proprietary shared-nothing compute engine, and storage on a blob store (Amazon S3; “in principle any type of blob store would suffice”). Compute nodes cache hot table data on local SSD, so once caches warm, performance “approaches or even exceeds that of a pure shared-nothing system.” They name the result the multi-cluster, shared-data architecture. This is the storage-compute-disaggregation axis the spoke now tracks. See snowflake for the architecture itself.

What it adds to the spoke

This is the first source describing the cloud-warehouse end of the spectrum, distinct from both duckdb (embedded) and clickhouse (a shared-nothing distributed engine — the very architecture this paper argues against for the cloud). It supplies the third axis in synthesis: not just embedded↔distributed, but whether storage and compute are coupled (DuckDB, classic ClickHouse) or disaggregated (Snowflake; and where modern ClickHouse Cloud / BigQuery / Databricks are heading).

Provenance note

Ingested from the CMU 15-721 course mirror of the SIGMOD 2016 paper (stable academic copy). The official DOI is 10.1145/2882903.2903741; Snowflake also hosts the PDF. Figures (queries/day, petabytes, the “15h on 4 nodes → 2h on 32 nodes” elasticity example) are the authors’ 2016 figures — dated snapshots, not current scale.

Related

snowflake · storage-compute-disaggregation · analytical-databases · clickhouse · duckdb · synthesis · snowflake-computing