Client-side load balancing

CSLB — the caller picks the destination itself. Instead of sending a request to a shared proxy that decides where it goes, each client process holds a view of the available endpoints and routes in-process: one network hop instead of two, no shared component in the middle to degrade, saturate, or blame.

The trade is ownership. A shared proxy is somebody else’s problem; a client-side balancer is a piece of distributed-systems machinery running inside every caller, and every caller now carries its staleness, its Kubernetes API watch, its RBAC, and its CPU cost. (zalando-cslb-1m-rps)

The parts

• Endpoint discovery. The client needs the live set of destinations — on kubernetes, the EndpointSlices API. Watch over poll: polling at fleet scale is a known way to hurt the control plane (zalando-cslb-1m-rps).
• Placement. Usually consistent hashing onto a ring of virtual nodes, so the same key lands on the same endpoint and per-endpoint caches stay warm; adding or removing an endpoint moves only its share of keys. The same primitive that partitions data in sharded stores, used here to partition traffic.
• Load signal. What “busy” means. In-flight request count is the cheap answer and a poor one — it can’t tell a fast endpoint with many requests from a slow one with few. Zalando switched to occupancy: accumulated request duration ÷ window, i.e. seconds of work per second, which is Little’s Law read as a utilization number (zalando-cslb-1m-rps).
• Bounded load. A cap that stops hashing from overloading a hot endpoint: if the hashed target is above its share, walk the ring until one isn’t, up to a hop limit.
• Warm-up. New endpoints get traffic on a ramp, not all at once, so a cold cache doesn’t answer full production load (zalando-cslb-1m-rps: a ^2.5 curve over 30s, one per scale event).

Where the primitives fight

Cache locality, zone affinity, and load spreading all want to place the same request in different places. Consistent hashing is what makes locality possible and is exactly what bounded-load walking and AZ affinity must break to do their jobs. Zalando shipped locality and bounded load, and paused zone affinity when its cost behaviour turned out to be unpredictable (zalando-cslb-1m-rps).

Where it sits

The service-mesh answer to the same problem is a sidecar proxy (Envoy fed by xDS) — client-side in topology, still a separate process. Zalando skipped the mesh entirely and put the balancer in the application process, which is the older library-style approach (Finagle, gRPC’s built-in balancers). It’s the routing sibling of container-scheduling: both are placement decisions, one for requests and one for workloads, and both get faster by giving up a globally consistent view.

Related

zalando-cslb-1m-rps · kubernetes · container-scheduling · modal-1m-sandboxes · observability · platform-ops