Design an outgoing webhook delivery service

The naive design is one shared queue and a pool of workers that POST and wait. Then a big merchant's endpoint starts taking 10 s to time out, every worker ends up waiting on it, and payment webhooks for 200k other merchants stall. This is head-of-line blocking and it is the main way these systems fail. Fix it at three levels. First, queue per endpoint (or per endpoint shard): workers pull from many queues fairly, for example round-robin across queues that have ready work, so a backlog on one endpoint does not sit in front of others. Second, a per-endpoint concurrency cap stored in Redis (in-flight counter with a short lease): if endpoint X already has 50 requests in flight, its jobs wait while workers serve other endpoints. Third, a circuit breaker per endpoint: after N consecutive failures stop sending, park jobs, and probe occasionally. Use async, non-blocking HTTP clients so a waiting request costs a few KB of memory, not a thread. Finally, separate pools for very large merchants or for slow endpoints (p50 latency above 2 s), the bulkhead pattern, so their volume cannot consume the shared pool.

The receiver must be able to prove the request came from us and is fresh. Each endpoint has a random 32-byte secret. The worker computes HMAC-SHA256(secret, webhook_id + '.' + timestamp + '.' + raw_body) and sends it in a signature header along with the id and timestamp. This is the open Standard Webhooks scheme; Stripe's is similar (it signs timestamp + body). The receiver recomputes the HMAC over the raw bytes (not re-serialised JSON), compares in constant time, and rejects timestamps older than about 5 minutes, which stops captured requests being replayed later. Rotation: when the merchant rotates, keep the old secret valid for 24 hours and send two signatures in the header (v1 with the new secret, v1 with the old), so the receiver can deploy the new secret whenever it likes without failed deliveries. Secrets are stored encrypted with a KMS key and decrypted only inside workers. For higher-security customers, offer asymmetric signatures (Ed25519) so the receiver holds only a public key, and optionally mTLS. Never put secrets or full payloads in our logs.

A webhook service is a machine that makes HTTP requests to any URL a customer types, from inside our network. Without protection, a malicious merchant registers http://169.254.169.254/latest/meta-data/ and reads cloud credentials, or targets internal admin services. Defences: require HTTPS on port 443 (allow other ports only by review); resolve DNS ourselves, reject private, loopback, link-local and multicast ranges (10/8, 172.16/12, 192.168/16, 127/8, 169.254/16, IPv6 equivalents), then connect to that exact IP so a DNS rebinding answer cannot switch it after the check. Do not follow redirects (or re-validate every hop). Run delivery workers in a separate network segment whose egress goes through a proxy (for example Smokescreen, which Stripe built for this) with no route to internal services. Cap response body reads at a few KB and enforce total timeouts so an endpoint cannot hold a connection open by trickling bytes. These details are rarely mentioned by candidates and immediately signal real production experience.

We cannot deliver exactly once. If the merchant's server processes the event and the connection drops before we read the 200, we will retry and they will see it twice. So the contract is at-least-once: every delivery carries a stable webhook-id (the same across retries and manual replays) and the documentation tells receivers to dedupe on it, for example with a unique constraint. Ordering is also not guaranteed: retries and parallel delivery mean refund.processed can arrive before payment.captured. Trying to enforce strict per-object ordering would mean one failed event blocks all later events for that merchant, which is worse. Instead each payload carries created timestamps and object versions, and we recommend receivers treat the webhook as a notification and fetch the current object from our API when order matters (the 'thin event' pattern). Internally we still get effectively-once fan-out: the delivery id is hash(event_id, endpoint_id), so if the fan-out consumer crashes and Kafka redelivers the event, the upsert creates no duplicate jobs.

Fat events carry the full object, which is convenient but can be stale on arrival and leaks more data if the endpoint is compromised. Thin events carry only id and type; the receiver fetches the latest state from our API. Offer fat by default with object versions, and recommend fetch-on-receive for money flows.

Kafka partitions cannot be created per endpoint (200k endpoints), and a slow message blocks its partition. So Kafka is used for the event stream and fan-out, while delivery jobs live in a store that supports many logical queues and delays: Redis streams or sorted sets, SQS with message groups, or a Postgres/Cassandra table polled with SKIP LOCKED-style leases.

Managed services such as Svix or Hookdeck already solve signing, retries and dashboards. For a payments company webhooks are part of the product, so building gives control over isolation, cost and data residency (RBI localisation for Indian payment data). For an internal tool, buying is usually right.

Longer retries (Stripe retries for up to 3 days) recover from weekend outages but keep backlog and storage longer. Shorter schedules simplify operations but push recovery onto merchants. Three days plus manual replay and a reconciliation API is a good balance.

Six hours of parked deliveries (say 2M) all become due. Drain at the endpoint's concurrency cap with ramp-up (start at 5 concurrent, double while success rate stays high), so we do not take their server down again.

Platform-level events with thousands of subscribers (partners, plugins) multiply volume. Fan out in batches, write jobs in bulk, and cap per-merchant subscriptions.

300M rows/day. Store in a columnar store (ClickHouse) partitioned by day and ordered by endpoint_id, time, so per-endpoint queries scan a small slice.

Jobs are leased with a visibility timeout of ~30 s. If a worker dies, the lease expires and another worker retries. The merchant might see a duplicate, which the stable webhook-id handles.

The deliveries table is the source of truth with next_attempt_at; a sweeper rebuilds the Redis schedule from rows where status = pending. Retries are delayed by minutes, not lost.

Product services keep writing to their outbox; events wait in Kafka (7-day retention). First-attempt latency SLO is breached and alerts fire, but nothing is lost; fan-out scales out to catch up.

Events stay in the outbox and Kafka; on recovery, delivery resumes with backlog draining at controlled rates. Merchants can also call a list-events API to reconcile anything they missed.