Design incoming webhook processing and reconciliation

Providers time out quickly (often 5-10 s) and retry on anything that is not a 2xx. If the handler confirms the order, calls the inventory service, sends an email and maybe runs an LLM fraud check inside the request, a slow dependency turns into provider timeouts, which turn into retries, which multiply load at exactly the wrong time. So the request does only three things: verify the signature over the raw body, insert the raw event, return 200. The insert is the durability point: once it commits we own the event. Everything else is async. Storing the raw payload (not just the parsed fields) is important: when a bug in processing is found, we fix the code and replay the stored events, which is impossible if we only kept what the old code extracted. The worker side uses Postgres as a queue with SELECT ... FOR UPDATE SKIP LOCKED, which is simple and transactional at a few thousand events per second; above that, put Kafka between receiver and workers, keyed by object_id so events for the same payment are processed in sequence. Return 400 only for invalid signatures; never return 5xx for 'business' failures, or the provider will retry forever.

Dedup by (provider, event_id) catches exact redelivery, but not everything: some providers send different event ids for the same state change, and events arrive out of order. The real protection is making the effect idempotent. Model each object as a state machine (created → authorized → captured → refunded) and apply changes with conditional updates: UPDATE ... SET status='captured' WHERE id=? AND status IN ('created','authorized'). Zero rows updated means the change was already applied or is stale, so it is a no-op. Where providers send an updated_at or version, also require it to be newer than the stored one. For important money events, do not trust the payload at all: fetch the object from the provider API and apply its current state. This 'webhook as a doorbell' approach makes order irrelevant, at the cost of an extra API call per event, which is fine at hundreds per second but must respect provider rate limits. Write the downstream event (PaymentConfirmed) through an outbox in the same transaction, so the state change and its announcement cannot diverge.

Webhooks are never complete. Providers have incidents, our endpoint can return 5xx during a bad deploy until the provider gives up, and some providers do not retry at all. Reconciliation turns 'usually correct' into 'eventually always correct'. Run a job every 5 minutes per provider (a cron, or a Temporal schedule for durable progress). It keeps a high-water mark, lists objects updated at the provider in [mark - overlap, now], and compares each with local state; any difference is fixed through the same transition code the webhook path uses, so there is one place that changes state. Overlap of ~10 minutes handles clock skew and provider indexing delay. Track drift (mismatches found per run) as a metric: it should be near zero; a rise means the webhook path is broken even if no error is logged. Add a daily pass that matches the provider's settlement report (amount, fees, UTR for Indian bank transfers) against captured payments, which catches money discrepancies the APIs do not show. Reconciliation must be rate-limited and paginated so it never competes with live payment calls on the provider's rate limit.

Compute the HMAC over the exact raw bytes received, before any JSON parsing; frameworks that parse and re-serialise change whitespace or key order and the signature will not match, which tempts developers to disable verification. Compare with a constant-time function. Check the timestamp header where the provider sends one and reject events older than ~5 minutes to prevent replays. Support two secrets during rotation. Keep per-provider verifier code behind one interface, since each provider differs (header names, encoding, some sign only the body). Do not log full payloads at info level: they contain names, phone numbers and card metadata, so apply PII redaction in the logging middleware. If a provider publishes source IP ranges, an allow-list at the WAF is a useful extra layer, but never a replacement for signatures.

Postgres with SKIP LOCKED keeps the raw event, status and retries in one transactional table, easy to query and replay; it is enough up to a few thousand events per second. Kafka scales further and preserves per-key order but needs separate retry topics and a DLQ topic. Start with Postgres; move when ingest or worker contention demands it.

Using the payload is fast and avoids provider rate limits. Fetching makes order and staleness irrelevant. Fetch for money-moving events, use the payload for low-risk status updates like courier location.

Every 5 minutes bounds the damage of a missed webhook to minutes but costs API quota. Hourly is cheaper but a customer waits an hour for an order confirmation. Pair a frequent narrow reconciliation with on-demand checks triggered by user-facing pending states.

A provider retry storm can produce thousands of events for the same few objects. Key processing by object_id (Kafka partition key, or advisory lock) so one object is handled by one worker at a time and conditional updates do not contend.

Fetch-on-receive plus reconciliation can exceed the provider's API quota during sales. Use a shared token bucket per provider account, prioritise live-processing fetches over reconciliation, and back off reconciliation when quota is tight.

45M rows/day. Daily partitions, drop or archive partitions older than 7 days to S3, keep only status indexes on recent partitions.

Providers retry with backoff, so most events arrive later. Whatever they gave up on is found by the reconciler on its next run. Users see delayed confirmations, not lost orders.

Raw payloads are stored, so after the fix an operator replays dead events for the affected window; idempotent transitions make replaying already-processed events safe.

Fetch fails, the event is retried with exponential backoff and jitter (1 s, 2 s, 4 s ... capped at 10 min, 8 attempts). If the provider stays down, events go to dead and are replayed or reconciled once it recovers.

Signature failures spike; alert immediately on the 400 rate per provider. Events are not lost because the provider retries, and reconciliation covers anything beyond its retry window.