Webhooks: sending and receiving reliably

Instead of a client polling 'is my payment done yet?', the provider sends an HTTP POST to a URL the client registered when the event happens: payment.captured, refund.processed, shipment.delivered, a pull request opened. Payment gateways, shipping partners, code hosts, CRMs and messaging platforms all use webhooks.

Webhooks travel over the public internet to servers the sender does not control. Receivers go down, deploy, time out or return errors. So senders retry, which makes delivery at-least-once: duplicates are normal. Retries and parallel delivery also mean events can arrive out of order, for example refund.processed before payment.captured. Some events never arrive at all, after the sender gives up.

Sending webhooks well means a durable delivery pipeline: an event is stored, queued per endpoint, signed, sent with a timeout, retried with backoff for hours or days, and endpoints that keep failing are paused, with logs and manual replay for customers.

Receiving webhooks well means: verify the signature, store the raw event, return 2xx quickly, process asynchronously and idempotently by event id, never trust arrival order, and run periodic reconciliation against the provider's API, because webhooks are a notification, not the source of truth.

1. 1Sender: when a business event commits, write a delivery record per subscribed endpoint (via the outbox, so no event is lost). Each event has a stable id that stays the same across retries.
2. 2Sender: sign each request with HMAC-SHA256 over timestamp + raw body using a per-endpoint secret, and send the event id, timestamp and signature in headers. Support secret rotation by signing with both old and new secrets for a while.
3. 3Sender: deliver from per-endpoint queues with a short timeout (for example 5-10 s) and a concurrency limit per endpoint, so one slow customer server cannot block deliveries to everyone else.
4. 4Sender: on non-2xx or timeout, retry with exponential backoff and jitter over hours or days (for example 1 min, 5 min, 30 min, 2 h ... up to 3 days). After repeated failures, mark the endpoint as failing, notify its owner and eventually disable it. Provide a dashboard with delivery logs and a 'resend' button.
5. 5Sender: protect yourself. Customer-supplied URLs can point to internal addresses, so resolve DNS and block private IP ranges and cloud metadata endpoints (SSRF protection); send from fixed egress IPs that customers can allow-list.
6. 6Receiver: read the raw body, verify the signature and timestamp before parsing, and reject anything older than about 5 minutes.
7. 7Receiver: insert the event into a table with a unique (provider, event_id), return 200 within a second or two, and process asynchronously with workers. A duplicate insert is a no-op that still returns 200.
8. 8Receiver: handle ordering by checking the object's status or version: ignore a payment.authorized arriving after payment.captured, or simply fetch the current object from the provider's API and act on that.
9. 9Receiver: reconcile. A scheduled job lists recent objects from the provider's API (or settlement reports) and fixes any order whose state does not match, catching webhooks that never arrived.

import hashlib, hmac, json, time

TOLERANCE_SECONDS = 300

def verify(raw_body: bytes, timestamp: str, signature_hex: str, secret: bytes) -> bool:
    # 1. Reject old requests to limit replay attacks.
    if abs(time.time() - int(timestamp)) > TOLERANCE_SECONDS:
        return False
    # 2. Sign the EXACT raw bytes received, never re-serialised JSON.
    signed = timestamp.encode() + b"." + raw_body
    expected = hmac.new(secret, signed, hashlib.sha256).hexdigest()
    # 3. Constant-time compare to avoid timing attacks.
    return hmac.compare_digest(expected, signature_hex)

def receive(request, db):
    if not verify(request.body, request.headers["X-Timestamp"],
                  request.headers["X-Signature"], SECRET):
        return 401
    event = json.loads(request.body)
    # Store and acknowledge fast; a worker processes it later.
    db.execute(
        "INSERT INTO webhook_events (provider, event_id, payload, status) "
        "VALUES ('pg', %s, %s, 'pending') ON CONFLICT (provider, event_id) DO NOTHING",
        (event["id"], request.body),
    )
    return 200   # duplicates also get 200, so the sender stops retrying