Distributed transactions: 2PC and sagas

A local database transaction is atomic: all of its writes commit or none do. Once an action spans two systems, for example deducting stock in the inventory service and charging money in the payment service, no single database can guarantee that. A distributed transaction is any protocol that tries to keep such a multi-system action consistent.

Two-phase commit (2PC) is the classic protocol. A coordinator asks every participant to 'prepare' (do the work, hold locks, promise it can commit), and only if all say yes does it tell them to 'commit'. It gives atomicity, but participants hold locks while waiting, and if the coordinator dies after prepare, they are blocked until it recovers.

A saga splits the action into a series of local transactions, each in one service, and defines a compensating transaction for each step (refund the payment, release the stock). If step 4 fails, steps 3, 2 and 1 are undone in reverse order. Sagas never hold locks across services, so they scale and survive partial outages, but the system is temporarily inconsistent and other users can see intermediate states.

Sagas are either orchestrated (a central workflow tells each service what to do, often built on a durable workflow engine such as Temporal) or choreographed (each service reacts to events from the previous one over Kafka). Both need idempotent steps and a reliable way to publish events, usually the transactional outbox.

1. 1First, try to avoid it: if the data that must change together can live in one database, use one local ACID transaction. Many 'distributed transaction' problems are really service boundaries drawn in the wrong place.
2. 22PC phase 1 (prepare): the coordinator sends PREPARE to all participants. Each writes the changes to its log, holds row locks and replies YES or NO.
3. 32PC phase 2 (commit): if every participant said YES, the coordinator durably logs COMMIT and sends it to all; otherwise it sends ABORT. Participants apply and release locks.
4. 42PC failure: if the coordinator crashes after participants voted YES, they cannot decide alone and keep holding locks until it returns. This blocking, plus locks held over network round trips, is why 2PC is used inside a database system (Spanner, CockroachDB, XA between a database and a broker) but rarely across independently owned microservices.
5. 5Saga design: list the steps in order, write a compensating action for each, and put steps that cannot be undone (sending an email, the final irreversible capture) last. Steps after which the saga must go forward are called pivot and retriable steps.
6. 6Orchestration: a saga orchestrator stores the saga's state durably, calls step 1, records the result, calls step 2, and on a permanent failure runs compensations in reverse. Temporal, AWS Step Functions or a state table in Postgres can hold this state.
7. 7Choreography: OrderCreated triggers inventory to reserve stock and emit StockReserved, which triggers payment, and so on. Fewer central components, but the flow is spread across services and harder to trace.
8. 8Reliability details: every step takes an idempotency key so retries do not double-charge; events are published via a transactional outbox so a database write and its event cannot diverge; and semantic locks (a 'PENDING' status) stop other requests from acting on half-finished data.

# Orchestrated saga, Temporal-style (simplified)
@workflow.defn
class CheckoutSaga:
    @workflow.run
    async def run(self, order: Order) -> str:
        undo = []
        try:
            await activity(reserve_stock, order, key=f"{order.id}-stock")
            undo.append(release_stock)
            await activity(debit_wallet, order, key=f"{order.id}-debit")
            undo.append(refund_wallet)
            await activity(confirm_order, order, key=f"{order.id}-confirm")
            return "CONFIRMED"
        except PermanentFailure:
            for step in reversed(undo):        # compensate in reverse order
                await activity(step, order, key=f"{order.id}-{step.__name__}")
            await activity(cancel_order, order, key=f"{order.id}-cancel")
            return "CANCELLED"