Storage-compute disaggregation

The architectural choice of handling storage and compute as two loosely coupled, independently scalable services rather than binding them together on the same nodes. Named and argued for in the Snowflake SIGMOD 2016 paper, it’s the third axis this spoke tracks — orthogonal to the embedded↔distributed one.

The contrast: shared-nothing

The dominant high-performance-warehouse architecture is shared-nothing: every query node owns its local disk, tables are horizontally partitioned, each node is responsible for its own rows. It scales well and runs on commodity hardware — but it couples compute capacity to storage capacity. In the cloud that coupling bites three ways the paper lays out: a single hardware config can’t suit both bulk-load and complex-query workloads; resize/failure forces data reshuffles on the same nodes doing query work; and online upgrades are hard when everything is tightly coupled and assumed homogeneous.

The disaggregated alternative

Put the durable data on a blob store (S3, Azure Blob, GCS) and run stateless compute clusters over it, each caching hot data locally. Then you can:

• scale compute and storage independently (and pay for each separately);
• spin compute up and down on demand, down to zero when idle;
• run many isolated compute clusters over one shared copy of the data;
• upgrade and tolerate failures without reshuffling base data.

The cost is network distance to storage, paid down by local SSD caching — warm caches recover, and sometimes beat, shared-nothing performance.

Where the spoke’s engines sit

• Disaggregated: snowflake (the canonical case). Modern ClickHouse Cloud, BigQuery (Dremel + Colossus), and Databricks (lakehouse on object storage) are the same family — unsourced here, a named open question.
• Coupled / shared-nothing: classic clickhouse (MergeTree on local disk per node). Storage-compute separation is precisely the axis on which clickhouse (coupled) and snowflake (disaggregated) differ, even though both are “distributed.”
• Single-box: duckdb — embedded, so storage and compute are the same process by definition; the disaggregation question doesn’t arise until you put DuckDB over object storage (e.g. DuckLake), which starts to blur the line.

Related

snowflake · snowflake-elastic-data-warehouse · analytical-databases · clickhouse · duckdb · synthesis