Design an OTP service at scale

Use a cryptographically secure generator (crypto/rand, SecureRandom), never Math.random or a time-based counter. Six digits is the norm; with at most 5 verify attempts per OTP the guessing chance is 5 in a million. Store the code as HMAC-SHA256 with a secret pepper kept in KMS, not as plaintext and not as a plain hash: a bare SHA-256 of a 6-digit code is reversed by trying all million values in milliseconds, so the secret key is what protects it if Redis data leaks. Bind the HMAC input to request_id and purpose so a login OTP cannot confirm a payment. Verification must be atomic: a Lua script in Redis reads the record, checks attempts, compares, increments attempts on failure, and deletes on success in one step; otherwise two parallel verify calls can both succeed or exceed the attempt limit. Compare in constant time. Return a generic error for wrong code and expired code to avoid leaking state, but show attempts left for usability. On success, return a short-lived signed token (JWT or opaque, 2 minutes) that the calling service checks, rather than the calling service trusting a boolean over the network.

Brute force on one OTP is handled by attempt limits, but an attacker can request a new OTP and get 5 fresh guesses, so limit sends per phone (e.g. 3 per 15 minutes, 10 per day) and lock verification for a phone after ~10 failures across OTPs. The more expensive attack is SMS pumping (artificially inflated traffic): fraudsters own or partner with premium-rate number ranges, mostly international, and use your OTP endpoint to send thousands of SMS they get paid for. Twitter reported losing tens of millions of dollars a year this way. Defences: allow-list the countries you actually serve; per-prefix and per-country velocity limits; watch the conversion rate (verified / sent) per prefix, since pumped numbers never verify; device attestation (Play Integrity, App Attest) or CAPTCHA before sending when risk is high; and hard spend caps per country per day. Also prevent account enumeration: the send API should respond the same whether or not the phone has an account.

Delivery success varies by aggregator, operator, region and time of day, so use two or three aggregators and route each message to the one with the best recent success rate and latency for the destination operator, measured from delivery receipts over the last few minutes. A circuit breaker drops an aggregator whose submit errors or receipt delays spike. OTPs go on a dedicated priority lane with reserved provider throughput (TPS), so a marketing campaign on the shared notification platform can never delay them. In India, TRAI's DLT system requires a registered entity, sender header and content template; the operator scrubs the message against the template, so the rendered text must match exactly, including the autofill hash line, or it is silently dropped. OTP and other service-implicit messages can be sent at any hour, unlike promotional ones. Fallbacks: WhatsApp authentication templates (often cheaper and very reliable in India) and voice calls for users with SMS issues. On resend, keep the same code so a slow first SMS remains valid.

Success of an OTP system is measured by how many users get through, not how many messages were sent. Track the funnel per tenant, purpose, country, operator, channel and provider: requested, accepted by provider, delivered (receipt), verified, and time between each step. Typical healthy delivery-to-verify conversion for login is high; a drop for one operator points at a delivery problem, while a drop for one prefix with high volume points at fraud. Autofill moves conversion and speed the most: Android's SMS Retriever API needs the message to end with an 11-character hash of the app's signing key and no SMS permission; the web WebOTP API needs a final line like '@example.com #123456'. Both remove typing errors and resends, which also cuts cost. Store this funnel in an analytics store (ClickHouse) fed by events from the service and from delivery-receipt webhooks, and alert on burn rate against an SLO such as '99% of OTPs verified-or-expired within 60 s'.

OTP state is short-lived and hot, with TTLs and atomic scripts, which fits Redis perfectly. Losing it means users request a new code, an acceptable failure. Use a replicated Redis cluster; keep the audit trail in a durable store separately.

SMS is universal in India and required by many flows, but vulnerable to SIM swap and phishing. Offer passkeys, TOTP or in-app device binding (as UPI apps do) for high-risk actions, and use SMS OTP as one signal among several.

Reusing the code means a late first SMS still works and avoids confusion; a new code limits exposure but invalidates messages already in flight. Reuse within the TTL, with a cap on total sends.

If the risk service is down, failing closed blocks all logins; failing open invites abuse. Fail open with tightened static per-phone and per-IP limits in the OTP service itself, and alert.

Aggregators cap throughput per account. Spread across providers and accounts, reserve OTP TPS, and pre-agree burst limits before known events.

Rate-limit keys for one big IP (carrier-grade NAT puts many mobile users behind one IP) get hot. Keep IP limits loose and lean on device and phone limits; shard counters with hash tags.

Rejected guesses still cost Redis calls. Rate-limit verify per request_id, phone and IP at the gateway, and cache 'locked' states locally for a few seconds.

Circuit breaker opens on submit errors or missing receipts; traffic shifts to the secondary within seconds. Users who already got 'sent' can resend, which uses the new route.

Replica is promoted in seconds; OTPs written in the last moment may be lost, so those users see 'expired, resend'. Rate-limit counters may reset, so keep conservative static limits as a backstop.

Receipts show one operator lagging; routing shifts to WhatsApp or voice for that operator's users, and the app shows the alternative channel earlier.