Delivery guarantees and effectively-once

A delivery guarantee describes what happens to a message when something fails between sender and receiver. At-most-once: send and forget; a crash can lose the message, but it is never processed twice. At-least-once: retry until acknowledged; nothing is lost, but a lost acknowledgement causes a duplicate. Exactly-once: each message affects the result exactly one time.

Exactly-once delivery over an unreliable network is impossible in general: if the sender does not get an acknowledgement, it cannot know whether the message or the acknowledgement was lost. What is achievable is exactly-once effect (often called effectively-once): deliver at least once, and make the receiver's processing idempotent so a duplicate changes nothing.

Kafka's 'exactly-once semantics' (EOS) is real but scoped. The idempotent producer removes duplicates caused by producer retries within a partition, and transactions let a consume-transform-produce job atomically write output records and commit input offsets. Consumers using isolation.level=read_committed then never see aborted writes. This is exactly-once inside Kafka. The moment your consumer charges a card, sends an SMS or writes to Postgres, Kafka cannot roll that back, so you are back to at-least-once plus idempotency.

Idempotency is therefore the core skill: give every message or request a unique id, and make the side effect conditional on that id not having been applied (a dedupe table in the same transaction, a unique constraint, an upsert keyed by a natural id, or a version check).

1. 1Producer side: retry sends on timeout, with a stable message id generated once (UUID or business id), so retries carry the same id.
2. 2Broker side: persist and replicate before acknowledging (Kafka acks=all with min.insync.replicas=2; SQS and RabbitMQ durable queues with publisher confirms). Enable the idempotent producer in Kafka so broker-level retries do not duplicate records.
3. 3Consumer side: process first, then acknowledge or commit the offset. A crash in between causes redelivery, never loss.
4. 4Make processing idempotent: in the same database transaction as the side effect, insert the event id into a processed_events table with a unique key. If the insert conflicts, skip the side effect.
5. 5Where possible use naturally idempotent operations: SET status = 'shipped' WHERE order_id = 7 AND status = 'packed' (a state transition), an upsert by business key, or 'set balance to X at version V' with optimistic concurrency, rather than 'add 100'.
6. 6For external calls (payment gateways, SMS, email), pass an idempotency key the provider honours (Stripe's Idempotency-Key header, a merchant order id for UPI or card payments). If the provider has no such key, record intent before calling and reconcile uncertain outcomes by querying status.
7. 7For pure Kafka-to-Kafka stream processing, use Kafka transactions (Kafka Streams processing.guarantee=exactly_once_v2, or Flink's two-phase-commit Kafka sink) so output and offsets commit atomically.
8. 8Bound the dedupe store: keep processed ids for longer than the maximum redelivery window (for example 7 days, matching topic retention), then expire them.

def handle(event, db):
    # One local transaction: dedupe record + side effect commit together.
    with db.transaction():
        inserted = db.execute(
            "INSERT INTO processed_events (consumer, event_id) VALUES (%s, %s) "
            "ON CONFLICT DO NOTHING",
            ("wallet-credit", event["event_id"]),
        ).rowcount
        if inserted == 0:
            return  # duplicate delivery: already applied, do nothing
        db.execute(
            "UPDATE wallets SET balance = balance + %s WHERE user_id = %s",
            (event["amount"], event["user_id"]),
        )
    # Commit the Kafka offset only after the DB transaction succeeded.
    # A crash before this line means redelivery, which the dedupe row absorbs.