Concurrency, backpressure and load shedding

Concurrency is how many requests are being worked on at once. By Little's Law, concurrency = arrival rate x latency. At 1,000 requests per second and 200 ms latency you hold 200 requests in flight. If a downstream slows to 2 seconds, the same traffic now holds 2,000 in flight: threads, memory, sockets and database connections run out even though traffic did not change.

Backpressure is a signal flowing upstream that says 'slow down'. It can be explicit (HTTP 429 or 503 with Retry-After, a gRPC RESOURCE_EXHAUSTED status, a bounded queue that blocks the producer, TCP flow control, Reactive Streams demand) or implicit (a caller with a concurrency limit that stops sending when its slots are full).

Load shedding is the server deciding to reject some work so the rest succeeds. Rejecting a request costs microseconds; accepting it and timing out after 10 seconds wastes the full cost and still fails. Good shedding happens early (at the edge or on admission), cheaply, and by priority: drop analytics and recommendations before checkout and payments.

The failure these tools prevent is congestion collapse: queues grow without limit, every request waits longer than its caller's timeout, callers retry, load multiplies, and the system does lots of work while successfully serving almost nothing. Bounded queues, timeouts, retry budgets and shedding keep 'goodput' (successful, useful responses) high under overload.

1. 1Bound everything: every thread pool, connection pool, in-memory queue and channel has a maximum size. An unbounded queue just converts overload into memory exhaustion and huge latency.
2. 2Set timeouts everywhere and propagate deadlines: if the client gives up after 3 seconds, downstream calls should be cancelled at 3 seconds too (context deadlines in Go, gRPC deadlines). Work for a caller who has left is pure waste.
3. 3Admission control: cap in-flight requests per instance (a semaphore). When full, reject immediately with 503 and Retry-After rather than queueing. Size the cap from Little's Law and load tests.
4. 4Make the cap adaptive when load varies: algorithms like Netflix's concurrency-limits (inspired by TCP congestion control) or Envoy's adaptive concurrency filter lower the limit when latency rises above the measured baseline and raise it when latency is healthy.
5. 5Shed by priority: tag requests with a criticality (critical, default, sheddable). Under pressure, reject sheddable first. Shed at the edge or gateway when possible, before work is done.
6. 6Prefer fresh work over stale work: drop requests that have already waited longer than their deadline; some systems serve the queue LIFO under overload or use CoDel-style queues that cap time in queue, so the requests served still have a chance to succeed.
7. 7Control retries: exponential backoff with jitter, a retry budget (for example retries at most 10% of requests), and no retries on 429/503 from a shedding server except after Retry-After. Use circuit breakers to stop calling a dependency that is failing.
8. 8For queues and streams, backpressure means consumers pull at their own pace; the producer side either blocks, gets rejected when a bounded buffer is full, or the backlog is absorbed by a durable log like Kafka while consumers scale on lag.
9. 9Degrade gracefully: serve cached or simplified responses (no personalisation, fewer results) instead of errors when downstreams are slow.

// Admission control: cap in-flight work and fail fast when full.
var slots = make(chan struct{}, 2000) // max concurrent requests per instance

func Admit(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        select {
        case slots <- struct{}{}:
            defer func() { <-slots }()
            // Bound the time this request may use downstream.
            ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
            defer cancel()
            next.ServeHTTP(w, r.WithContext(ctx))
        default:
            // Full: reject now instead of queueing forever.
            w.Header().Set("Retry-After", "1")
            http.Error(w, "overloaded", http.StatusServiceUnavailable)
        }
    })
}