Design friends nearby

The first instinct is to put everyone in a geohash or H3 index and query 'users near me', then filter to friends. That is wrong here. The candidate set is already small and known: my ~400 friends. A geo query returns thousands of strangers in a dense area like Bengaluru and throws almost all away. Instead, give every active user a pub/sub channel. When I come online, my WebSocket server subscribes to the channels of my online friends. When a friend publishes, the message reaches only servers that care, and the server does a cheap distance check per local subscriber. Cost is proportional to friendships between online users, not to population density. A geo index (geohash prefixes per cell channel) becomes the right choice only when the audience is unknown, such as 'people nearby' discovery; then you subscribe to the channels of your cell and its 8 neighbours. Saying both, and why you picked one, is the key signal.

At ~13M deliveries/s the bottleneck is CPU on the pub/sub nodes, not memory. A single Redis node can push a few hundred thousand messages/s, so plan for ~100 nodes with headroom. Channels are sharded with consistent hashing: channel loc:{user_id} → node via a hash ring stored in etcd, which every WebSocket server watches. Note that Redis Cluster's own sharded pub/sub (SPUBLISH, Redis 7) also works, but classic PUBLISH in Redis Cluster broadcasts to every node and does not scale. Resizing is the delicate part: pub/sub is fire-and-forget, so moving a channel loses messages for a moment. Do it in a quiet hour: update the ring, servers re-subscribe moved channels on the new node, and unsubscribe from the old one. Losing one or two 30-second updates is acceptable for this product, which is why stateless pub/sub is a good fit. Do not use Kafka for this fan-out: millions of tiny per-user topics are not what Kafka is for, and its per-partition ordering and retention add cost without value.

Cost scales linearly with update frequency: going from 30 s to 5 s multiplies location QPS (333k to 2M/s) and fan-out (13M to 80M/s) by 6. Pick 30 s while the screen is open and stop or slow down (e.g. every 5-15 minutes using the OS significant-change API) in the background. The client can also skip sending when it has moved less than ~50 m since the last update, which removes most updates from people sitting at home or office; the server refreshes TTL with a cheap heartbeat instead. On the receive side, push only meaningful changes: a friend entering or leaving the radius, or a distance change of more than ~10%. Rounding distance ('about 2 km away') reduces both privacy risk and push volume. On Tier-2/3 networks with patchy 4G, the client should reconnect with jittered backoff and send its latest position on reconnect, not a backlog of old points.

Location is sensitive data, and mistakes here are reputational incidents. Make sharing opt-in with an explicit audience, and enforce it on the server at fan-out time, not only when subscribing, because settings can change between subscribe and publish. Keep a small cached settings map per user (enabled, audience list version) and check it per delivery. Coarsen location before it leaves the server: snap to a grid of ~100-500 m or add bounded noise, and show 'about 2 km' instead of exact coordinates, so repeated readings cannot triangulate a home. Block and unfriend events must take effect within seconds: deliver them via Kafka to the servers holding either user and unsubscribe immediately. Location TTL means a user who closed the app vanishes after 10 minutes. History has a short retention enforced with Cassandra TTLs, and a deletion request removes the partition. Under India's DPDP Act you need clear consent and purpose limitation, so do not reuse this history for ads without separate consent.

Polling '/friends nearby' every 30 s from 10M clients means 333k reads/s, each doing an MGET of ~400 keys: 130M key reads/s, mostly unchanged data. Pub/sub moves only actual changes and keeps the cache read path for the initial snapshot. Polling is acceptable for an MVP with a few thousand users; at scale, push wins on cost and latency.

Redis pub/sub: in-memory, tiny channels, no persistence, which matches 'latest position only' semantics. Kafka: durable and replayable, but per-user topics are impossible and per-partition routing would make every server read everyone's updates. A custom routing tier (like Facebook's internal systems) can be more efficient but is a big build. Choose Redis pub/sub and say it is acceptable to lose an update.

Exact coordinates enable a map view but increase privacy risk and push volume. Approximate distance bands ('<1 km', '1-3 km', '3-5 km') hide precise location and need pushes only on band changes. Default to approximate; offer a precise live share only as an explicit, time-limited action.

A user with 5,000 friends online makes one publish fan out to thousands of servers' subscribers. Since one message goes to each subscribing server, not each user, the cost is bounded by the number of WebSocket servers (~200), which is fine. Cap the feature to a friend-count limit or show only close friends for very large graphs.

At 100M concurrent users, fan-out reaches ~130M deliveries/s. Reduce work before adding nodes: skip unchanged locations on the client, push only band changes, raise the interval to 60 s in the background, and route with sharded pub/sub so each node handles only its channels. Then scale the ring horizontally.

After a WebSocket server deploy or crash, 100k clients reconnect at once and each triggers friend-list reads and ~40 subscribes. Use jittered client backoff, drain servers slowly during deploys, and cache friend lists for online users so reconnects do not hammer the friends service.

Channels on that node stop delivering. WebSocket servers detect the dropped connection, the ring in etcd is updated to a replacement node, and servers re-subscribe their affected channels. Users see friends' positions freeze for up to a minute; the 10-minute TTL in the cache keeps the snapshot valid meanwhile.

Positions on that shard are gone, so some friends temporarily disappear from snapshots. Within 30 s every active user sends a fresh update and the cache refills. No replication is strictly needed, though a replica makes failover quieter.

Kafka buffers events; the live path does not depend on history writes, so users notice nothing. Alert on consumer lag and scale the history writers or Cassandra.