Scaling to 1 million concurrent sandboxes in seconds (Modal)

Modal’s engineering blog on rebuilding the scheduler behind its Sandboxes so a customer can go from zero to a million concurrent sandboxes in under a minute. The old system topped out at 50,000 concurrent sandboxes per customer; the rewrite claims no practical ceiling, a median start under half a second, scheduling latency in tens of milliseconds, and creation throughput of tens of thousands of sandboxes per second. T3 — a vendor blog about the vendor’s own product, and every number is self-reported with no reproducible method published. The architecture is the durable part; the benchmarks are marketing until someone measures them independently.

The design move: drop global consistency

The rewrite’s premise is that a strongly consistent central scheduler is the thing that caps burst scale, so Modal “traded global consistency for scalability and performance” on the sandbox-creation path and left no data store in it at all. Four pieces:

Horizontally scaled scheduling servers. A fleet handles creation requests concurrently instead of one serialized scheduler, each running a fast algorithm over in-memory cached data — the post’s own framing is that this makes scheduling “more like load balancing than traditional container scheduling.”
Workers as the source of truth. Workers periodically publish their own state to a Redis stream rather than a central durable datastore writing it down for them. Scheduling servers consume that state asynchronously, so they act on a slightly stale view by design.
Direct RPC creation. A scheduling server picks a worker from its in-memory view and contacts the worker directly; the worker accepts or rejects based on the resources it actually has. Rejection, not consensus, is what keeps the stale view honest. The whole path is two network hops and one cheap CPU operation.
Batched control messages. Fleet operations are batched under data-oriented design rather than paying O(sandboxes) RPCs.

The Kubernetes critique

The post argues directly against the Kubernetes scheduler as a model for this workload: its O(n × p) algorithm (n nodes × p pods) is serialized by default, etcd becomes a bottleneck as pod writes land, and node heartbeats impose a baseline O(nodes) write load before any work happens. Modal’s counter is asynchronous state propagation — the coordination cost that Kubernetes pays for a consistent cluster view is the cost Modal refuses to pay. See container-scheduling for where this sits against the spoke’s other scheduling sources.

Where the bottlenecks actually were

Two are worth keeping, because they are the parts that no architecture diagram predicts:

The kernel, not the scheduler. Creating containers fast enough caused contention on the Linux kernel’s rtnl lock during container networking setup, stalling startups by tens of seconds. They “had to change our container networking setup for sandboxes” to stop workers falling over. The scheduler was fine; the kernel’s network-namespace path was the wall — the same shape as Meta’s kernel-level find, one layer over.
Redis, deliberately unsharded. Every worker publishes to a single Redis stream. Load testing put that as viable “until well over 100,000 workers,” with sharding into multiple streams as the escape hatch. A known ceiling they chose not to engineer around yet.

Cost and status

Months of work across the backend, including reimplementing every Sandbox feature and all Sandbox observability, plus changes to worker management and the container runtime. (The post notes four engineers spent eight days prototyping intensively in Miami Beach — the rewrite’s origin, not its total cost.) It ships in Beta, opt-in with “one simple change to your code.” Stated use cases: reinforcement learning (the workload that actually wants millions of concurrent sandboxes), background coding agents, and traffic bursts in agent-facing systems.

Cross-spoke context

cloud-wiki — Modal as a rented sandbox substrate competing with E2B and AWS’s Firecracker-backed [[aws-lambda-microvms]] is that spoke’s story (its synthesis already names Modal in an open question on where agent-written code gets executed). The offering is cloud-wiki’s; the scheduler behind it is here.
agentic-tooling-wiki — the agents that consume these sandboxes, and harness-internal sandboxing ([[cloud-run-sandboxes]]), sit there. Modal is the substrate under that layer, not the layer itself.

modal · compute-sandbox · container-scheduling · kubernetes · serverless · scaling-to-1m-lambda · meta-sched-ext-ads · platform-ops