Load balancing
In one line
A load balancer spreads incoming requests across many healthy servers using an algorithm such as round robin or least connections, so the service scales horizontally and survives server failures.
What it is
A load balancer sits between clients and a pool of backend servers and decides which server handles each connection or request. Clients see one address; behind it you can add, remove or replace servers freely.
Layer 4 load balancers route TCP or UDP connections by IP and port without reading the content, which is very fast. Layer 7 load balancers understand HTTP, so they can route by path, host or header, terminate TLS, retry requests and add headers, at a higher CPU cost.
Health checks are as important as the algorithm: the load balancer regularly probes each server (for example GET /health every 5 seconds) and stops sending traffic to servers that fail.
Why interviewers ask
Load balancing is in nearly every system design answer, because it is how you scale beyond one server and remove single points of failure. Interviewers probe which algorithm you would pick, how you handle sessions and failures, L4 versus L7, and whether the load balancer itself becomes a bottleneck.
How it works
- 1Clients resolve the service name to the load balancer's address (often a virtual IP or several IPs via DNS).
- 2The load balancer keeps a list of backends and their health. Servers that fail several checks in a row are marked down and get no new traffic; they are added back after passing again.
- 3For each new connection (L4) or request (L7), the algorithm picks a healthy backend: round robin cycles through servers, weighted round robin gives bigger servers more, least connections picks the server with the fewest active connections, and hashing on a key (client IP, user id) sends the same key to the same server.
- 4The load balancer forwards the traffic. An L7 balancer typically terminates TLS, opens or reuses its own connection to the backend and adds headers like X-Forwarded-For with the client's real IP.
- 5During deploys, servers are drained: they get no new requests but finish in-flight ones before shutting down, so users see no errors.
Worked example · Least connections with a failing server
- Pool: A, B, C. Active connections: A=4, B=2, C=3.
- Request 1 arrives: least connections picks B (2). Now A=4, B=3, C=3.
- Request 2 arrives: B and C tie at 3; tie broken by order, pick B. Now A=4, B=4, C=3.
- Request 3: pick C (3). Now A=4, B=4, C=4.
- C fails two health checks in a row (every 5 seconds, so within about 10 seconds) and is marked down. Its connections drop; clients retry.
- Request 4: only A and B are eligible, both at 4; pick A. Plain round robin would have kept sending every third request to C until it was marked down, and would have ignored that A already had more work.
Picture it
A host at a busy restaurant seats each new group at the table of the waiter who currently has the fewest tables (least connections), skips any waiter who is on a break (failed health check), and keeps regulars with their favourite waiter (sticky sessions).
Key patterns
Round robin and weighted round robin
When: Similar servers and similar-cost requests
How: Rotate through servers in order; give bigger servers higher weights.
Least connections or least response time
When: Requests vary a lot in duration or connections are long-lived
How: Send each new request to the server with the fewest active connections or lowest latency.
Hashing and consistent hashing
When: You want the same key to hit the same server (caches, sharded state)
How: Hash the client IP or user id; consistent hashing means adding or removing a server moves only about 1/n of the keys.
Sticky sessions
When: Legacy apps that keep session state in memory
How: Pin a client to one server with a cookie. Prefer making servers stateless, because stickiness causes uneven load and lost sessions on failure.
Redundant load balancers
When: Avoiding the balancer as a single point of failure
How: Run active-passive or active-active pairs with a floating IP, or use a managed cloud balancer that is already redundant.
Real-world use
Cloud load balancers
AWS offers the Application Load Balancer (L7, routes by path and host, terminates TLS) and the Network Load Balancer (L4, very high throughput and static IPs); Google Cloud's global load balancer uses one anycast IP worldwide.
NGINX and HAProxy
Widely used software load balancers in front of web apps, supporting round robin, least connections, IP hash, health checks and TLS termination.
Kubernetes Services and Ingress
A Kubernetes Service spreads traffic across pod replicas at L4, and an Ingress controller (often NGINX or Envoy) adds L7 routing such as /api to one service and /web to another.
Client-side load balancing
gRPC clients and service meshes like Envoy pick a backend themselves from a service registry, avoiding an extra hop; this matters for long-lived HTTP/2 connections that an L4 balancer would pin to one server.
In a project
Your single-server API handles 500 requests per second, but a marketing launch is expected to bring 5,000, and the one server is also a single point of failure.
- Make the API stateless by moving sessions into Redis or signed tokens, so any server can handle any request.
- Put an L7 load balancer in front of an auto-scaling group of 4 to 20 servers, using least outstanding requests because response times vary.
- Configure a /health endpoint that checks the server can reach its database, with checks every 5 seconds and 2 failures to mark down.
- Enable connection draining (30 seconds) and rolling deploys, and spread servers across at least two availability zones.
- Watch per-backend latency and error rates to spot one bad server.
Impact: The service handles 10x traffic by adding servers instead of buying a bigger one, survives the loss of a server or a whole zone without user-visible errors, and deploys no longer cause downtime.
Trade-offs
Use it when
- More traffic than one server can handle, or high availability is required
- Rolling deploys and auto-scaling without downtime
- L7: routing by path or host, TLS termination, retries; L4: raw TCP/UDP at very high throughput
Avoid it when
- Sticky sessions as a substitute for stateless design
- Plain round robin when request costs vary widely
- An extra L7 hop for latency-critical internal calls where client-side balancing is simpler
Cost: Adds one network hop (well under a millisecond inside a data centre). Choosing a backend is O(1) for round robin, O(log n) or O(n) for least connections depending on the data structure, and O(log n) for consistent hashing on a sorted ring.
Common mistakes
- Forgetting health checks, so traffic keeps going to dead servers.
- Making the load balancer itself a single point of failure.
- Keeping session state in server memory and then relying on sticky sessions.
- Assuming an L4 balancer spreads requests evenly for HTTP/2 or gRPC; it balances connections, and one long-lived connection can carry all of a client's requests to one server.
- Losing the client's real IP behind the balancer instead of reading X-Forwarded-For or the PROXY protocol.
Interview questions
Quick revision
- Load balancer: one front address, many backends, health checks.
- L4: IP and port, fast. L7: HTTP-aware, path and host routing, TLS termination.
- Algorithms: round robin, weighted, least connections, least response time, hash.
- Consistent hashing moves about 1/n of keys when a server changes.
- Prefer stateless servers over sticky sessions.
- Connection draining for zero-downtime deploys.
- Make the balancer itself redundant; spread backends across zones.
Check yourself
1. Requests take anywhere from 10 ms to 5 seconds. Which algorithm usually spreads load best?
2. Which feature needs a layer 7 load balancer?
3. What is the main downside of sticky sessions?
4. With consistent hashing over 10 servers, roughly what share of keys move when one server is added?
Saved on this device. Create a free account to keep it on every device.
Practise this with Sam
A real mock interview, out loud, with an honest scorecard. Free to start.