Design Tatkal train booking (IRCTC)

The worst design lets every request race to the database at 10:00:00; the winners are whoever has the fastest script, and the database melts. Instead, decouple accepting a request from processing it. The intake tier does only cheap, stateless checks and appends to a durable log, so it scales horizontally to ~100k requests/s. The order in the log per train is the fairness order: arrival at our edge, with no advantage from hammering because one user can have only one active request per train and date. Requests before the server's opening time are rejected, not queued, so clock tricks do not work. Bots are the real fairness threat: enforce login with OTP on verified accounts, captcha at login (not at 10:00, when it would slow real users), device and behaviour scoring at the CDN, per-account and per-IP limits, and block agents for the first 30 minutes. Show honest queue status. Load from hammering the status endpoint is absorbed by serving it from Redis with client-side backoff hints (Retry-After). Pre-window work (login, form fill, search) spreads load over 30 minutes instead of one second.

A train-date-class has at most a few hundred Tatkal berths and receives thousands of requests per second at open. Putting that on a database row (UPDATE available = available - n WHERE available >= n) creates a hot row where every transaction waits on one lock. Since the data is small, use a single-writer allocator: one process owns a train-date-class partition, holds its berth bitsets in memory and processes requests sequentially from the Kafka partition. Each allocation takes microseconds, so even 10k requests/s for one train is easy, and there is no lock contention. Durability comes from the log: the allocator writes its decisions to an output topic and PNR rows, and snapshots state periodically. If it crashes, a standby takes over the partition (Kafka consumer group rebalancing, or a lease in etcd), loads the last snapshot and replays results since then; inputs already decided are skipped by request_id. Many partitions spread thousands of trains across ~50-100 allocator instances. This is the same pattern stock exchanges use for order books: serialise per instrument, parallelise across instruments.

Train seats are not like cinema seats: a berth from New Delhi to Patna can also be sold from Patna to Howrah. Model each berth's journey as a bitset over the train's ordered stops (a train with 30 stops needs 29 segment bits). A request for stations i to j needs a berth whose bits i..j-1 are all free; allocation sets those bits. Searching is a scan over the class's berths with bitwise AND, which is fast for a few hundred berths. Quotas complicate it: Tatkal berths are tagged, and some quotas apply only to certain segments (for example pooled or remote-location quotas). Prefer allocations that keep long stretches free (best-fit on the segment) to reduce fragmentation, and honour preferences (lower berth for senior citizens) as a soft score. RAC (two passengers sharing a side-lower berth) and waitlist are counters per class with ordered lists; when a confirmed passenger cancels or fails to pay, the allocator upgrades RAC 1 to confirmed and WL 1 to RAC in order. Availability shown in search is a summary computed from these bitsets and published every second.

Allocation before payment is fairer (the user knows the outcome fast) but ties up berths for users who never pay. Bound it with a short payment window (~5 minutes) after allocation. The payment service creates a payment with idempotency key = PNR; UPI and netbanking results arrive by callback, sometimes late. Three cases: paid in time → confirm; failed or no response by the deadline → release berths through the allocator's log (as an event, so ordering stays consistent) and offer the user nothing more; paid after release → refund automatically, because the berths may already belong to someone else. A reconciliation job compares bank settlement files with PNRs every few minutes and fixes mismatches. The historical alternative used by IRCTC is pay-first then book, with automatic refunds when booking fails; it protects inventory from non-payers but leads to many refunds at Tatkal time. Either way, money movement must be idempotent and auditable, and refunds must be automatic and fast, because 'money deducted, ticket not booked' is the most common Tatkal complaint.

Synchronous booking holds a connection open through allocation and payment setup; at 50k requests/s that is millions of waiting connections and timeouts that users retry, making it worse. Accepting into a durable queue and returning 202 keeps the intake fast and makes retries harmless. The cost is a polling UX, which users accept if status is clear.

Database row locks are simple and durable but serialise on the hot row with network round trips per transaction. An in-memory single writer gives microsecond allocations and natural ordering, at the cost of building snapshot, replay and failover. Given the extreme contention per train, the single writer is worth it.

Allocate-then-pay: fast, clear outcome, but berths can be held by non-payers for minutes. Pay-then-allocate: no unpaid holds, but many refunds and anxious users. A short payment window with immediate re-allocation to the next request balances both.

If everyone logs in at 09:59, the auth service becomes the bottleneck. Encourage early login (sessions valid for the window), rate limit per account, cache user eligibility in the session, and scale auth separately from booking.

Two million clients polling every second create 2M requests/s. Serve status from Redis via a lightweight endpoint, return a Retry-After hint that grows with queue size, and push results by SMS or app notification as well.

Thousands of payments per second right after allocation can exceed bank and gateway limits. Spread across gateways and banks, and extend deadlines automatically for payments in progress at the gateway, so bank slowness does not release a paid berth.

A standby takes the partition within seconds, loads the latest snapshot and replays decisions from the output topic. Requests not yet decided are processed in order. Users see a few seconds of delay, never a double allocation, because each request_id is decided once.

Replication factor 3 with acks=all and min.insync.replicas=2 means accepted requests survive a broker loss. Leadership moves in seconds; intake retries appends with the same request_id.

If allocations must sync to an older central reservation system, publish them asynchronously and keep the allocator as the authority for the Tatkal window, with the central system updated through a reliable outbox. Pause new allocations only if sync lag exceeds a safety limit.