Design an event-sourced order pipeline

Each order is a stream of events with a strictly increasing version. To handle a command (cancel), load the snapshot if any, fold the remaining events into current state, check business rules (cannot cancel after shipping), and append the new event at version n+1. The primary key (order_id, version) makes this optimistic concurrency control: if two commands race, both read version 6, both try to write 7, and the database rejects one, which reloads and re-checks. No locks are held while thinking. Postgres works well as the event store at this scale (an indexed append per event, tens of thousands per second on good hardware, more with hash sharding by order_id); EventStoreDB or DynamoDB with a conditional put on (pk=order_id, sk=version) are alternatives. Events are facts in past tense and are never updated or deleted. If something was wrong, append a compensating event (PaymentRefunded, OrderCorrected). This is what gives audit and support a complete history: 'why was this order cancelled?' is answered by reading the stream, with actor and causation ids in the metadata.

If the command handler writes the event to Postgres and then publishes to Kafka, a crash between the two leaves an event that nobody hears about; publishing first risks announcing an event that was never saved. The fix is the transactional outbox: the event and an outbox row commit in one database transaction, and a separate relay publishes outbox rows to Kafka and marks them sent. With CDC (Debezium reading the Postgres WAL) the event table itself can be the outbox. The relay can publish a row twice if it crashes after sending but before marking, so delivery into Kafka is at-least-once; consumers dedupe on event_id. Key every message by order_id: Kafka only orders messages within a partition, so a random key lets PaymentCaptured overtake OrderPlaced. Producer settings matter: acks=all, enable.idempotence=true, and topics with replication factor 3 and min.insync.replicas=2, so an acknowledged event survives a broker loss. Propagate the W3C traceparent in Kafka headers so one trace spans API, relay and every consumer.

Kafka's exactly-once (idempotent producer plus transactions) only covers read-process-write loops that stay inside Kafka. A consumer that charges a card or writes to Postgres gets at-least-once delivery, so correctness comes from idempotent processing. Pattern: in one database transaction, insert event_id into processed_events and apply the side effect; a duplicate violates the primary key and the whole thing is skipped. Commit the Kafka offset only after that transaction commits. For external side effects that cannot share our transaction, such as the payment gateway, pass a deterministic idempotency key (order_id or event_id) so the provider deduplicates. Ordering within a partition plus version checks in projections (apply only if event.version is newer than last_version) make replays safe. The failure modes to name: committing the offset before the side effect loses events; committing after but without idempotency duplicates them. Use cooperative-sticky partition assignment so scaling consumers does not pause the whole group on every rebalance.

The fraud scorer calls an LLM that may take 30 s or fail 20% of the time. If it retried inline, one stuck order would block every later order in its partition. So failures are moved aside: after an immediate retry, publish the event to orders.retry.1m, then orders.retry.10m, each consumed by a worker that waits until the message is due; after the last attempt, publish to orders.dlq with the error, stack trace and attempt count in headers. The main partition keeps moving. Because the LLM is non-deterministic, store its output (score, reasons, model and prompt version) in the FraudScored event itself. Replays then reuse the recorded decision instead of re-asking the model and getting a different answer, which is a core event-sourcing rule: record the result of non-deterministic calls. Add a business fallback: if no score arrives within 2 minutes, apply a rules-only score so orders are not stuck. Scale the scorer on consumer lag (KEDA), but cap replicas at the partition count and at the LLM rate limit, or you only produce 429s faster.

Event sourcing gives full history, replayable projections and natural audit, at the cost of more complexity: schema evolution, eventual consistency in reads, and harder ad-hoc queries. Many teams get most of the benefit with a normal orders table plus outbox events. Choose event sourcing when the history itself has business value, as in orders, payments and ledgers.

Projections lag ~1 s, so a user may refresh and not see the order yet. Mitigate by returning the new state in the command response, reading your own writes from the event store for the details page right after a command, or waiting on the projection version in the client.

Events live forever, so old shapes must stay readable. Use a schema registry with backward-compatible changes (add optional fields), a schema_version on every event, and upcasters that convert old versions to the current shape on read. Never rename or reuse fields in place.

Consumers reacting to events (choreography) stay decoupled but the end-to-end flow is hard to see. For multi-step flows with compensation (reserve, pay, ship, refund on failure), an orchestrator such as Temporal or a saga service makes the flow explicit. A common split: events for facts and fan-out, orchestration for the checkout saga.

Keying by order_id spreads load well; problems come when keying by merchant_id or user_id. Keep order_id as the key and do merchant-level aggregation downstream.

At 10× (250k events/s) one Postgres primary is not enough. Hash-shard streams by order_id across several primaries (Citus or app-level), or move to DynamoDB with conditional writes.

Rebuilding from years of history takes too long. Use snapshots, keep compacted Kafka topics of latest state for some views, and parallelise rebuilds across partitions.

Commands still succeed because they only need the event store; the outbox grows and the relay catches up when Kafka returns. Projections and downstream actions are delayed, so the UI shows 'processing'.

An event the consumer cannot parse would block the partition forever. After retries it goes to the DLQ with error details, the partition moves on, and an alert fires. After a fix, replay from the DLQ.

Fraud scoring events pile up in retry topics; the 2-minute fallback applies rules-only scoring so orders keep flowing. When the provider recovers, the scorer drains the backlog at its rate limit.

Never edit history. Append a correcting event and fix the code; projections that consumed the bad event either handle the correction or are rebuilt.