Design a notification system (push, SMS, email)

The most common failure of notification systems is head-of-line blocking: a 50M-message campaign fills the queue and a login OTP waits behind it for 20 minutes. Fix it structurally, not with priority fields inside one queue. Use separate topics per priority and channel (critical-sms, transactional-push, marketing-push, marketing-email) with separate consumer groups and reserved worker capacity for the critical ones. Rate limits to providers are also split: reserve, for example, 30% of the SMS provider's throughput for OTPs, and let marketing use the rest only when free. Apply admission control at the API: campaign traffic is throttled at the scheduler, so it enters the pipeline at a steady rate instead of 50M at once. Monitor queue lag per topic with SLOs (critical lag under 1 s, marketing lag under 30 min). This is the same bulkhead pattern used in other systems, and naming it explicitly shows you have seen real incidents. Also separate per tenant: one internal team's bug that sends 10M messages should hit its own quota, not everyone's.

Users hate duplicates ('Your order is delivered' three times) and losing an OTP is worse. Between our service and providers, true exactly-once is impossible: if a call to the SMS provider times out, we do not know whether it was sent. So the design is at-least-once with dedup at each step. Producers send an idempotency key; the API's conditional insert makes repeated calls safe. Kafka consumers may redeliver after a crash, so each channel job has a deterministic job ID (notification_id + channel) and workers check a sent-status record before calling the provider, updating it after success. For the ambiguous timeout, we retry because a rare duplicate SMS is better than a lost OTP, but many providers accept a client reference ID that lets them dedup too. For push, FCM collapse keys and APNs apns-collapse-id replace earlier notifications with the same key on the device, which hides duplicates. A per-user dedup window (same template and data within 10 minutes) catches upstream bugs.

External providers are the least reliable part. Each provider has throughput limits (FCM allows high rates but throttles bursts to a single device; SMS aggregators have TPS limits per account; SES has a per-second send rate), so workers use a distributed token bucket per provider in Redis. Retries use exponential backoff with jitter and classify errors: 4xx such as invalid token or invalid number are permanent, so drop and clean up; 429 and 5xx are retryable. After N failures a job goes to a dead-letter queue with alerting. For SMS in India, use two or three aggregators and route by live success rate per telecom operator (Jio, Airtel, Vi), because delivery quality varies by operator and time; a circuit breaker opens when a provider's error rate spikes and traffic shifts to the backup. Delivery receipts arrive later by webhook; verify their signatures, process them idempotently, and map provider message IDs back to our notification_id. Track cost per channel, since SMS and WhatsApp messages cost real money and push is free.

Every send is filtered by user preferences, which makes the preference lookup the hottest read. Cache preferences in Redis with explicit invalidation on update, and keep a small default policy per category (transactional cannot be turned off, promotions can). Quiet hours: for marketing, if it is 11 pm in the user's timezone, defer to the next allowed window instead of dropping; transactional messages ignore quiet hours. Frequency caps (for example at most 3 promotional pushes per day) use Redis counters with daily TTL. India adds hard rules: commercial SMS must use DLT-registered sender IDs and templates, the rendered text must match the registered template or the operator rejects it; promotional SMS cannot be sent at night (TRAI rules restrict promotional messages to daytime hours) and must respect the national Do Not Disturb registry. Email needs one-click unsubscribe and proper SPF, DKIM and DMARC to stay out of spam, which large senders like Gmail now require. Building these checks into the platform means every product team gets compliance for free.

Synchronous sends would tie callers to provider latency and outages. Accepting after a durable write and processing async decouples producers. The cost is callers must check status or subscribe to callbacks if they care about delivery.

Kafka gives high throughput, replay and ordering per key, good for campaigns and analytics. SQS or RabbitMQ give per-message acks, delays and DLQs out of the box, convenient for retries. Many teams use Kafka for ingestion and a job queue with delayed retries per channel.

SMS is reliable but costs money; push is free but not guaranteed. Push first, SMS fallback for important unacknowledged messages saves large costs while keeping critical messages delivered.

A central platform adds a dependency, but gives consistent preferences, compliance, cost tracking and provider failover. Per-team integrations duplicate this work and spam users.

50M lookups in 30 minutes. Batch reads (MGET 500 users at a time), pre-join preferences into the segment export, and cache aggressively.

We cannot push faster than providers allow. Spread campaign sends over the window, use multiple accounts or providers, and shape traffic with token buckets.

Delivery receipts for a big campaign arrive in bursts. Accept webhooks quickly into a queue and process async; aggregate stats in a stream job.

Circuit breaker opens on error rate, traffic shifts to the secondary provider, OTP latency stays within SLO. Pending retries go to the new provider.

Kafka offsets are committed after jobs complete; redelivered jobs check sent-status first, so most duplicates are suppressed.

A kill switch per campaign and per template stops sends within seconds; campaigns go through a canary on 1% of the segment first.