Design an API gateway

The gateway adds latency to every API call, so the rule is: nothing on the request path waits on a remote store unless it must. Routes are compiled into an in-memory radix tree, so matching is microseconds. JWTs are validated locally: the gateway caches the identity provider's public keys (JWKS), refreshes them every few minutes and on an unknown key ID, and checks signature, expiry and scopes. Opaque API keys are hashed and looked up in a local cache backed by Postgres, with a TTL of ~60 s and push-based revocation events for fast blocking. Plugins run in a fixed order and must be cheap; anything slow (fraud scoring, content scanning) belongs in a service, not in the gateway. Use an event-loop proxy (Envoy, NGINX/OpenResty) with keep-alive connection pools to upstreams so TCP and TLS handshakes are not paid per request. With this, the gateway's own cost stays ~1-2 ms at p50. Measure it separately: log total latency and upstream latency, and alert on the difference.

A limit like '1,000 requests/min per API key' is global, but traffic for one key spreads across 50 nodes. Options: (1) Call Redis on every request with an atomic Lua script (sliding-window or token bucket). Accurate, but it adds ~0.5-1 ms and a Redis dependency to every call; at 500k req/s that is a big Redis cluster. (2) Pure local limits: divide the limit by node count. No network cost, but uneven load balancing makes it unfair. (3) Hybrid, which most large gateways use: each node keeps a local token bucket and syncs its consumption to Redis every ~100 ms or every N requests, getting back the global count and adjusting its local share. Accuracy is within a few percent, and Redis load drops by 10-100x. Use sliding-window counters for fairness at window edges. If Redis is unreachable, fail open with local-only limits (protect availability) for normal APIs, but fail closed for abuse-sensitive routes like OTP or login. Return 429 with Retry-After and X-RateLimit-Remaining headers.

The gateway sees every service failure first, and it must not amplify it. Each route has a timeout lower than the client's (for example 2 s for the gateway vs 5 s on the app), so a slow service frees gateway resources. Retries only for idempotent methods (GET, or POST with an Idempotency-Key), at most 1-2 attempts, with jittered backoff and a retry budget (retries ≤ 10-20% of requests) so retries cannot triple the load during an outage. Outlier detection ejects instances that return consecutive 5xx. Circuit breakers cap concurrent requests and pending queue per upstream: when Payments is slow, its connection pool fills and new Payments requests fail fast with 503, while Orders traffic is unaffected. This is bulkheading per upstream. For a festive sale, priority-based load shedding drops low-value traffic (recommendations) before checkout. Hedged requests can cut tail latency for idempotent reads but double load, so use them only for a few critical paths.

Split the system so the data plane keeps working when the control plane is down. The control plane owns the config database, validation, RBAC (which team may change which routes), audit logs, and compilation of snapshots. Data-plane nodes subscribe over a streaming API (Envoy's xDS is the standard) and get versioned, complete snapshots, not individual diffs, so a node that reconnects simply takes the latest version. Nodes keep the last applied config on disk; if the control plane is unreachable they keep serving with it. Changes roll out progressively with health gates. Validation catches route conflicts (two teams claiming /v1/users) and dangerous settings (timeout 0, retries 10). This split is also how you offer the gateway as a platform: hundreds of teams self-serve routes without a central team editing NGINX files and doing reloads.

One gateway gives one place for auth and limits but becomes a shared bottleneck in process and blast radius. Larger companies run several gateway clusters (public, partner, internal) built on one platform, plus BFF services per client type for aggregation. Keep business logic out of the gateway either way.

Managed (AWS API Gateway, Apigee) is fast to start and includes keys and quotas, but costs per request (expensive at 500k req/s), adds latency and limits customisation. Self-hosted Envoy or Kong gives control and low cost at scale but needs a platform team. Choose managed for small scale, self-hosted at large scale.

If the rate limiter or auth key cache is unavailable, failing open keeps the business running but removes protection; failing closed protects backends but turns a small dependency failure into a full outage. Decide per route: auth fails closed (with cached keys as a buffer), most rate limits fail open.

A sale start brings millions of new connections; full TLS handshakes are CPU heavy. Use TLS 1.3, session resumption, ECDSA certificates, keep-alive, and pre-scale gateway nodes before known events instead of relying on reactive autoscaling.

One very large API consumer makes its rate-limit key hot on one Redis shard. Use the hybrid local-bucket approach so the key is touched only on sync, or split the key into N sub-keys and sum them.

The L4 load balancer health check removes it within seconds. Nodes are stateless (state is in config snapshots and Redis), so others take the load. Keep N+2 capacity per zone.

Gateways keep serving with their last-known-good snapshot. Only config changes are blocked. Emergency changes (block an abusive key) can go through a small, separately deployed denylist channel.

Timeouts, circuit breakers and per-upstream connection limits isolate it. The gateway returns fast 503s for that route and protects its own threads and memory so other routes are unaffected.