Object storage as a coordination primitive

Using an object store (S3 and its API-compatible relatives) not as a place to park bytes but as the agreement mechanism between processes that need a shared, changing view of some state. The phrase comes from a practitioner reading of Canva’s design: S3 as “more than an object store”, built on “immutable objects and conditional writes as coordination primitives” (canva-session-revocation-infoq).

The two primitives that make it work

• Conditional PUT — write this object only if it is still the version I read. That is compare-and-swap, and compare-and-swap is enough to build optimistic concurrency control with no lock service anywhere. Writers race, losers re-read and retry, and correctness never depends on who wins (canva-session-revocations-at-scale).
• Conditional GET — give me this object only if it changed. Readers can poll continuously at close to zero cost, which is what turns a store into a propagation channel rather than a thing you have to invalidate.

Around those, two design habits recur. Immutability plus chunking: cut the state into ranges (Canva uses 30-minute slices of a 12-hour window) so all but the newest object is frozen and cacheable forever, and only the tail is contended. And a leader that isn’t required: Canva runs ZooKeeper leader election to keep writers from colliding, but the conditional PUT already guarantees correctness, so the election is a throughput optimization whose failure degrades performance rather than integrity (canva-session-revocations-at-scale).

Why reach for it

The alternative to a shared database is usually another distributed system to operate — a Redis cluster, more read replicas, a consensus service. Object storage is the one piece of infrastructure a cloud team already has, already durable, already effectively unbounded in read fan-out, and already somebody else’s on-call. Canva names avoiding that extra system as the reason for the choice.

The load shape is the real prize. A database read path is sized by how many clients ask; an object store read path is sized by how often the data changes, because unchanged chunks are free to poll. That is precisely the property Canva reports as the outcome: database load stopped scaling with gateway instance count (canva-session-revocations-at-scale).

What it is bad at

Latency and granularity. Propagation is measured in the seconds-to-minutes a poll cycle takes, not in the microseconds a cache lookup takes, and the unit of change is a whole object — small frequent writes to a hot object turn into a retry storm on one key. So it fits state that is append-mostly, bounded, and tolerant of a stale read: deny-lists, config, manifests, catalogue metadata. It does not fit anything that needs read-your-writes.

Where it sits here

This is the same trade the spoke has now met four times, and the variant that is easiest to miss: the shared component on the hot path is not deleted, it is replaced by one with a better load curve. modal-1m-sandboxes removes the data store from the creation path outright, zalando-cslb-1m-rps moves routing into the caller, and Canva keeps a shared thing in the middle but picks one whose cost tracks writes instead of readers (session-revocation). The pattern also rhymes with what openobserve does one layer up, running queries against Parquet in place: the same bet that object storage has become fast and cheap enough to sit somewhere it used to be too slow for.

Related

canva-session-revocations-at-scale · canva-session-revocation-infoq · session-revocation · modal-1m-sandboxes · zalando-cslb-1m-rps · openobserve · platform-ops