Timeouts, retries and circuit breakers

In a distributed system, a slow dependency is more dangerous than a dead one. A dead service fails fast; a slow one holds your threads, connections and memory until your own service also stops responding. Timeouts, retries and circuit breakers are the three basic tools that keep one sick dependency from taking down everything that calls it.

A timeout is the maximum time you wait for a call. A retry repeats a call that failed with a transient error, such as a dropped connection or a 503. A circuit breaker watches the failure rate of a dependency and, when it is too high, fails calls immediately for a short time instead of sending them, then lets a few test calls through to see if the dependency has recovered.

The three work together. Without timeouts, retries never start because calls hang. Without limits on retries, a small outage turns into a retry storm that multiplies load exactly when the dependency is weakest. The circuit breaker is the safety valve that turns 'slow and failing' into 'fast and failing', which is something the caller can handle with a fallback.

1. 1Set a timeout on every outbound call: connect timeout (short, around 100 ms to 1 s inside a data centre) and request timeout. Base the request timeout on the dependency's measured latency, for example a bit above its p99.9, not on a round number like 30 seconds. Default HTTP clients in many languages have no timeout at all.
2. 2Use a deadline, not only per-hop timeouts. The edge sets a total budget (say 800 ms) and passes the remaining time to each downstream call, as gRPC does with deadline propagation. A service that sees only 20 ms left should fail fast instead of starting work nobody will wait for.
3. 3Retry only transient failures: connection errors, timeouts, 502, 503, 504 and 429 (respecting Retry-After). Do not retry 400, 401, 403 or 404; they will fail again.
4. 4Retry only operations that are safe to repeat. Reads are safe. Writes are safe only if they are idempotent, usually with an idempotency key the server stores, so a retried 'charge Rs 500' is not charged twice.
5. 5Use exponential backoff with jitter: wait base x 2^attempt, capped, and pick a random value in that range ('full jitter'). Without jitter, thousands of clients retry at the same instant and create synchronised waves of load.
6. 6Limit retries in two ways: a small per-call cap (2 or 3 attempts in total) and a retry budget per client, for example retries may add at most 10% extra traffic. Retry at one layer only, usually the one closest to the failing dependency; retrying at every layer multiplies load.
7. 7Wrap each dependency in a circuit breaker with three states. Closed: calls pass and failures are counted over a sliding window. Open: after the failure rate crosses a threshold (for example 50% of at least 20 calls in 10 seconds), calls fail immediately for a cool-down period. Half-open: after the cool-down, a few trial calls pass; success closes the circuit, failure opens it again.
8. 8Give every open circuit or exhausted retry a fallback: a cached value, a default, a degraded feature, or a clear error. A fallback is a product decision, so agree it with the product team before the outage.
9. 9Isolate resources with bulkheads: separate connection pools or concurrency limits per dependency, so a slow recommendation service cannot use up the threads the checkout path needs.
10. 10Measure it all: timeout counts, retry counts, breaker state changes and fallback rate per dependency should be metrics with alerts.

import random, time

RETRYABLE = {502, 503, 504, 429}

class CircuitOpen(Exception):
    pass

class Breaker:
    def __init__(self, threshold=0.5, min_calls=20, cooldown=30.0):
        self.threshold, self.min_calls, self.cooldown = threshold, min_calls, cooldown
        self.calls = self.failures = 0
        self.opened_at = None

    def before(self):
        if self.opened_at is not None:
            if time.monotonic() - self.opened_at < self.cooldown:
                raise CircuitOpen()          # fail fast, caller uses fallback
            self.opened_at = None            # half-open: let a trial call through
            self.calls = self.failures = 0

    def record(self, ok):
        self.calls += 1
        self.failures += 0 if ok else 1
        if self.calls >= self.min_calls and self.failures / self.calls >= self.threshold:
            self.opened_at = time.monotonic()

def call_with_retry(send, breaker, deadline, attempts=3, base=0.1, cap=2.0):
    for attempt in range(attempts):
        breaker.before()
        remaining = deadline - time.monotonic()
        if remaining <= 0:
            raise TimeoutError("deadline exceeded")
        try:
            status, body = send(timeout=min(remaining, 2.0))
        except (ConnectionError, TimeoutError):
            status, body = None, None
        ok = status is not None and status < 500 and status != 429
        breaker.record(ok)
        if ok:
            return status, body
        if status is not None and status not in RETRYABLE:
            return status, body              # non-retryable (e.g. 500): return as is
        # full jitter backoff, never beyond the deadline
        time.sleep(min(random.uniform(0, min(cap, base * 2 ** attempt)),
                       max(0, deadline - time.monotonic())))
    raise TimeoutError("retries exhausted")