Design a feature flag and config service

The most important design choice is that server-side flag checks never call the flag service. At ~10M evaluations per second, even a 1 ms remote call would add latency to every request and make the flag service a single point of failure for the whole company. Instead, each SDK downloads the full ruleset for its environment (a few MB), keeps it in memory, and evaluates rules as a pure function. Delivery then becomes a fan-out problem: a change written once must reach about 20,000 SDK instances quickly. Use a streaming relay tier: SDKs hold a long-lived SSE (or gRPC stream) connection; on connect they receive the full ruleset with a version number, then small patches. Relays subscribe to a change bus, so adding relay pods scales delivery. Patches carry versions, and an SDK that sees a gap (version 810 then 812) re-fetches the full snapshot. If the stream drops, SDKs reconnect with jittered backoff and poll the snapshot (with ETag) every 30-60 s as a fallback. On cold start with the service completely down, SDKs load the last snapshot from S3/CDN or a local file baked into the image, and finally fall back to the default value passed in code. The flag service can be fully down and production keeps working on the last known config.

A percentage rollout must be sticky (the same user always gets the same variant), consistent across services (checkout and the order page agree), and monotonic (going from 5% to 25% keeps the original 5% in). Do not use random numbers or store per-user assignments. Hash instead: bucket = murmur3(flag_salt + ':' + user_id) mod 10,000, giving buckets 0-9,999 with 0.01% granularity. Variant 'on' owns buckets 0 to (percent × 100 - 1). Raising the percent only extends the range, so existing users stay in. Each flag has its own salt; if all flags hashed only the user id, the same 1% of users would get every new feature at once and experiments would be correlated. For experiments that must be independent of a previous rollout, change the salt (re-randomise) on purpose. Choose the bucketing key carefully: user_id for logged-in features, device_id for logged-out users, or tenant_id/restaurant_id when a feature must apply to a whole business. Note that a user who logs in mid-session may switch buckets if you change keys, so define it per flag.

Mobile apps cannot receive the full ruleset: it would leak targeting data (list of beta users, unreleased feature names, enterprise customers) and could be large. So clients use server-side evaluation for one user: the app sends its context (user id, app version, city, platform) with a client-side key to an evaluation API at the edge, and receives only the final values. The app caches them on disk and uses them on next start, which matters on Tier-2/3 networks where the first request may take seconds. Responses can be cached at the CDN per context hash with a short TTL. Updates reach clients on app start, foreground and every 5-15 minutes, or via a streaming connection for web; for a kill switch you can send a silent push. Two mobile-specific traps: old app versions live for months, so every flag needs a safe default compiled into the binary; and a flag that changes UI mid-session can confuse users, so many apps 'freeze' flag values for the session and apply new ones on next launch, except kill switches.

Many large incidents come from config changes, not code deploys, because config often skips the testing that code gets. Treat flag changes like deploys: validation (type checks, a JSON schema for config flags, limits like 'timeout_ms between 50 and 10,000'), required reason and approval in prod, staged rollouts linked to SLO metrics with automatic halt and rollback, and a one-click revert to any previous version from the audit log. Keep kill switches simple and always available, without approvals. Limit blast radius by environment and by region (enable in one region first). Clean up: every flag has an owner and an expiry; the system flags stale ones (100% on for 30 days, or not evaluated for 14 days) and opens a ticket to remove the code path, because old flags become a combinatorial testing problem. Never put secrets in flags; ruleset snapshots are widely distributed.

Remote evaluation (call the service per check) is simple, keeps rules secret, and changes apply instantly, but adds latency and a hard runtime dependency. Local evaluation is fast and resilient but needs SDKs in every language, a delivery system, and accepts a few seconds of staleness. Use local for servers, remote (edge-cached) for untrusted clients.

Polling every 30 s is simple and robust behind any proxy, but changes take up to 30 s and 20,000 pollers create steady load. Streaming delivers in about a second with tiny load per change, but needs long-lived connections, reconnect logic and proxies configured not to buffer. Use streaming with polling as a fallback.

LaunchDarkly, Unleash, Flagsmith, ConfigCat and cloud services (AWS AppConfig) solve most of this. Building makes sense for very large companies that need deep integration (experiments, data localisation, cost at scale); otherwise use OpenFeature as the SDK interface so the backend can be swapped.

20,000 SDKs reconnecting at once all request the full ruleset. Use jittered backoff in SDKs, drain relays slowly during deploys, and serve full snapshots from a CDN/S3 with ETags rather than from Postgres.

A 'beta users' list of 2M ids inside the ruleset makes every SDK download huge. Store big lists separately, ship them as compact sets only to services that need them, or evaluate those flags server-side.

Server SDKs continue with in-memory rules; new pods start from the S3/CDN snapshot or a bundled file; worst case they use code defaults. Only flag changes are impossible during the outage, which is why the kill-switch path should also have a break-glass option (e.g. directly publish a signed snapshot).

Schema validation should have rejected it; if not, SLO-linked rollout halts the change in the first stage, and one-click revert restores the previous version within seconds. Postmortem adds a validation rule.

SDKs report their ruleset version in a heartbeat; the control plane shows how many instances are on old versions and SDKs with version gaps re-fetch the full snapshot automatically.