Design a stock trading platform (Zerodha Kite)

The classic bug: a user with ₹10,000 sends two ₹8,000 orders at the same moment from two devices, and both pass the check. The fix is to make 'check available margin, block it, create the order' one atomic step on one shard. Shard orders and the margin ledger by user_id so they live in the same Postgres database. In one transaction, run SELECT ... FOR UPDATE on the user's balance row (or an optimistic update: UPDATE balances SET blocked = blocked + 8000, version = version + 1 WHERE user_id = ? AND available - 8000 >= 0 AND version = ?), insert the ledger 'block' entry, and insert the order with status PENDING_SEND. Only after commit is the order sent to the exchange. Per-user row locks are fine because one user rarely sends more than a few orders a second, so there is no hot row across users. When the order is rejected or cancelled, a 'release' entry is written in the same way. Margin for F&O is harder: SPAN margin depends on the whole portfolio, and hedged positions need less margin. Precompute the user's current portfolio margin in RMS memory, and calculate the incremental margin for the new order; refresh SPAN parameter files that the clearing corporation publishes several times a day. Interviewers want to hear 'atomic block before send' and 'per-user serialisation', not a distributed lock across services.

The dangerous case is a timeout: OMS sent the order, the leased line dropped, and no ack arrived. Did the exchange accept it? If you mark it failed and the user retries, they may end up with two positions. If you assume success, money stays blocked. So the order goes to an UNKNOWN state, never to 'rejected'. Each exchange session uses sequence numbers; on reconnect, the gateway replays from the last sequence it confirmed and the exchange resends missed messages, which resolves most unknowns within seconds. The broker also sends a unique client order reference with each order, so it can query the exchange's order status by that reference. The UI shows 'status being confirmed' and blocks a duplicate order with the same details for a short window. Run each exchange session from a primary and a hot standby gateway; only one is logged in at a time because exchanges allow a set number of sessions per trading member. At end of day, compare the broker's order book and trade book with the exchange trade file and the clearing obligation file. Any mismatch is a break that ops must fix before next market open. Saying this clearly shows you understand that external systems are authoritative.

Ticks are high volume and only the latest value matters, so this is a pub/sub problem with conflation, not a queue problem. Do not put each tick through Kafka per user. Instead, market data handlers push updates to ticker nodes that each hold the full latest snapshot in memory (10k instruments × ~200 bytes is only a few MB). Each WebSocket edge node serves ~50k-100k connections and keeps, per instrument, the set of connections subscribed. Every 250 ms-1 s it walks the instruments that changed and writes one batched binary frame per connection. Conflation means a slow client on a 3G network in a Tier-3 town gets the latest price, not a growing backlog; if its send buffer stays full, drop intermediate updates and finally disconnect it. Use a compact binary format: a token, then fixed-width integers for price in paise, so a full-mode packet is a few dozen bytes. Separate the order update channel from ticks on the same socket with message type, and give order updates priority, since a missed fill notice is much worse than a missed tick. At 9:15 AM, millions reconnect at once; protect the edge with connection rate limits, jittered client reconnects and pre-scaled nodes, because autoscaling reacts too slowly for a spike that lasts minutes.

Traffic is not smooth: login and order load in the first five minutes after open can be 20-30x the daily average, and F&O expiry afternoons spike again. Plan capacity for the peak, not the mean, and scale up before 9:00 AM on a schedule. Pre-warm everything: load instrument master and SPAN files, warm connection pools, and preload each active user's balance and positions into RMS memory before market open. After-market orders (AMO) placed overnight are queued and released to the exchange right at open, so stagger them across gateway sessions and respect each session's exchange message rate limit; a token-bucket per session keeps you from being throttled by the exchange. Keep the order path short and synchronous only up to the gateway; everything else (positions, notifications, analytics) consumes Kafka asynchronously. If a downstream service is slow, the order path must not wait. Finally, shed non-critical load at peak: disable heavy reports and historical chart APIs for a few minutes, and keep order placement and cancellation as the protected path. A cancel must always get through, even when new orders are being rate limited.

Orders and margin need multi-row transactions (block money and create order together) and strong consistency, which Postgres gives on one shard. Cassandra or DynamoDB would scale writes easily but make the margin check a conditional write puzzle across items. At ~20k orders/s peak, 16-32 Postgres shards with replicas handle the load. Use Kafka and S3 for the high-volume event history, so Postgres holds only the current day's hot data.

Sending every exchange tick to every client is wasteful and breaks slow clients. Conflating to the latest value every 250 ms-1 s cuts output by 10x or more and is what a human eye can follow. The cost is that clients do not see every intermediate price, which is fine for retail UIs. Algo users who need every tick should use a separate paid feed or co-location, not the retail WebSocket.

Waiting for the exchange ack before replying makes the API latency depend on the exchange and on the leased line, and holds a request thread per order at peak. Replying once the order is durably stored and queued for the gateway keeps p99 low; the ack and fills come back over the WebSocket. The UI must then show a clear 'pending' state. Most brokers do this.

Many brokers start with a vendor OMS that is already certified by exchanges, because exchange certification and compliance are slow and costly. Building in-house gives control over latency, features like GTT and baskets, and cost at scale. In an interview, mention the choice and say you would build the user-facing services and wrap the vendor gateway behind a clean internal API either way.

At 10x users, millions of logins and WebSocket reconnects hit in the same minute. Fix: long-lived sessions issued the previous day where allowed, stateless signed tokens validated at the edge, connection admission control with retry-after, and reconnect jitter in the client. Pre-scale edge fleets on a schedule before 9:00 AM.

Exchanges cap messages per second per trading session. At 10x orders, the broker needs more sessions (more leased lines and exchange approvals) and must spread users across them. A per-session token bucket queues bursts briefly instead of getting throttled or disconnected by the exchange.

On expiry day, NIFTY and BANKNIFTY options near the money are subscribed by most users. Sharding ticker nodes by instrument would make one node hot. Instead, replicate the full snapshot on every edge node (it is small) so each node serves its own connections without a central hop.

Each exchange session has a backup line and a standby gateway. In-flight orders move to UNKNOWN; on reconnect, sequence-number recovery replays missed acks and fills. New orders are queued for a few seconds or rejected with a clear message. Cancellations get priority once the line is back.

Handlers subscribe to both A and B multicast streams and fill gaps from whichever has the packet; if both miss, request retransmission or a snapshot recovery. If prices go stale, the app marks them stale with a timestamp rather than showing old prices as live. The GTT engine pauses triggers on a stale instrument.

Because positions are built from Kafka events, a slow consumer means positions update late, but the order book (from OMS) is still correct. Kafka keeps the events; the consumer catches up idempotently using (exchange trade_id) as a dedupe key. Margin is updated from the same events, so RMS also relies on trade ids to stay idempotent.

Synchronous replication to a standby in the same region (with a second region for DR) means no committed order is lost; failover takes seconds. Orders for users on that shard are briefly rejected with a retry message. Never fail over to an async replica without checking for lost writes against the exchange order log.