Client-Side Load Balancing at a Million Requests Per Second

Zalando Engineering (2026-06), by Conor Gallagher, Senior Principal Engineer at zalando. How the Product Data Serving team took over 1M requests/sec of internal fan-out traffic off the shared cluster load balancer by moving the routing decision into the caller’s own process — an in-process client-side load balancer built on consistent hashing, Kubernetes endpoint watches, and an occupancy-based load signal.

The problem

The Product Read API (PRAPI) sent its internal traffic through Skipper, Zalando’s shared Kubernetes ingress/edge proxy (kubernetes). Two things made that hop expensive:

Fan-out multiplies exposure. One batch request unpacks into up to 100 downstream calls, so any Skipper degradation is felt ~100× per user request.
Shared infrastructure blurs attribution. When latency spiked, nobody could say whether PRAPI or Skipper owned it — the observability boundary ran straight through the shared component.

At peak, a million requests a second went through that single shared hop.

The mechanism

The CSLB runs inside the product-sets pods; Skipper stays for edge traffic only.

Hash ring. Reimplements Skipper’s xxHash64 consistent hash with 100 virtual nodes per endpoint — deliberately identical routing decisions, so the migration could be compared against the old path rather than merely trusted. Ring updates swap a single atomic reference (lock-free reads).
Endpoint discovery by watch, not poll. Polling EndpointSlices had previously caused control-plane incidents at scale; the CSLB uses a watch-based informer with a 2s debounce across scale events.
N-ring fade-in. A new pod does not get full traffic on arrival. Each scale event opens its own fade-in curve (^2.5 over 30s) so the pod warms its cache with the entries consistent hashing will actually send it.
Occupancy, not in-flight count, as the load signal. In-flight requests were abandoned in favour of accumulated request duration ÷ time window — “seconds of work per second” via Little’s Law — composited with in-flight and weighted by the endpoint’s latency relative to the cluster average (capped at 5×). Bounded-load then walks the ring past overloaded endpoints: p50 zero hops, p99 four hops against a 10-hop cap.
AZ-aware routing — built, then paused. Zone affinity was suppressed when a zone’s latency drifted >35% above baseline, but the feature was switched off before full rollout (see below).

Inspiration credited to Finagle for latency-weighted load; profiling via Java Flight Recorder, metrics via Micrometer. No service mesh — no Envoy, no xDS, no gRPC. Direct HTTP.

Results (Zalando-reported)

>1M req/s moved off Skipper; latency dropped and the daily spikes flattened to a steady baseline. Cache-hit ratios were identical through the canary, which is the check that the replicated hash ring worked.
Skipper fleet: 50+ pods → 8 (~$450/day → $110/day). Inter-AZ transfer (EC2-Other) fell sharply.
The occupancy signal removed another ~25% of pods (~$1,000/day) and let the HPA threshold rise from 50% → 65% CPU. Per-pod occupancy tightened from a 0.40–1.30 spread to 0.60–0.90.
Deploy pipeline: build 21 → 12 min; median deploy 289 → 128 min; worst case 4d 21h → ~2h.
No load-balancing incidents since the migration completed.

What they gave up

Owning the load balancer means owning ring staleness (mitigated by keeping the last-good ring), the Kubernetes watch and its RBAC, the on-call escalation, and the CPU the balancer burns in every caller. Two honest admissions carry more weight than the headline numbers:

Consistent hashing fights its neighbours. Cache locality, zone affinity, and bounded-load redistribution all want to place the same request differently; locality only survives if each partition has enough nodes to cover the hot product sets. Zone-aware routing was paused after trials turned up unpredictable DynamoDB cost trade-offs and bad interactions with scale-up fade-in — restart gated on a clearer cost model and team health.
Telemetry beat algorithms. Adding pod IP and node to error logs exposed recurring 2–3 second node freezes that had been invisible — Gallagher rates that more valuable than the load-balancing improvements themselves.
A slow pipeline is a reliability problem. Slow deploys push teams toward big batches and deferred changes; a fast pipeline with automated safeguards was judged safer than a slow one with manual gates.

Why it matters here

This is the spoke’s first source on load balancing as an owned component rather than inherited infrastructure, and it answers the standing quantification gap with money, pod counts, and pipeline times rather than adjectives. It also sits directly across from modal-1m-sandboxes: both delete a shared coordination point from the hot path and accept a stale, local view in exchange, but Modal does it for burst placement while Zalando does it for steady-state routing.

T2 — first-party engineering blog with specific mechanisms and figures, all self-reported and unreplicated.

client-side-load-balancing · zalando · kubernetes · observability · dora-metrics · site-reliability-engineering · modal-1m-sandboxes · container-scheduling · platform-ops