Design a payment system (Stripe-like)

Networks fail between every pair of systems, so every hop can be retried, and every retry must be safe. At our API: the merchant sends an Idempotency-Key; we insert it with a hash of the request before doing work. If the insert conflicts, we either return the stored response (same request) or reject (different body with same key). If the first request is still in progress, return 409 so the client retries later, never run it twice. Store the final response, including errors that are deterministic. Towards the processor: generate our own unique reference per payment attempt and send it; many processors deduplicate on it, and it allows a status lookup after a timeout. Inside our system: Kafka gives at-least-once delivery, so consumers are idempotent: the ledger poster uses (payment_id, event_type) as a unique key for journals, and the webhook dispatcher includes an event_id so merchants can deduplicate. This is effectively-once processing: messages may be delivered more than once, but effects happen once. State it precisely; claiming exactly-once delivery over the network is a red flag.

Model a payment as an explicit state machine with allowed transitions: created → processing → requires_action → authorized → captured → partially_refunded/refunded, and failed, canceled, and unknown. Each transition is a conditional update (WHERE status = expected AND version = v) inside a transaction that also writes the event to an outbox table, so state and events never diverge. The most important state is unknown: a timeout from the processor does not mean failure. Marking it failed and letting the merchant retry is how customers get charged twice. Instead, a recovery worker resolves unknown payments by querying the processor with our reference, with backoff, and finally the settlement file is the ground truth. Webhooks from processors can arrive out of order or duplicated; the state machine rejects transitions that go backwards and treats duplicates as no-ops. Keep a full transition history for support and audit. Timers matter too: authorisations expire (often after about 7 days for cards), so a scheduled job captures or voids them before expiry according to merchant settings.

The payments table says what happened to a payment; the ledger says where the money is. Keep them separate. In a double-entry ledger every movement is a journal with two or more lines, and debits equal credits in each journal. Accounts include merchant balance, platform fees, processor receivable, bank cash, refunds payable and reserves. A ₹1,000 capture with a 2% fee posts: debit processor receivable 1,000; credit merchant balance 980; credit fee revenue 20. When the processor settles, debit bank cash 1,000 and credit processor receivable 1,000. A refund reverses the relevant parts. The ledger is append-only: never update an entry; post a correcting journal. Balances are sums of lines, with periodic snapshots for speed. Integrity checks run continuously: every journal balanced, every captured payment has exactly one capture journal, the receivable account drains to zero after settlement. Use integer minor units and store the currency on every line. Write throughput of thousands of lines per second fits a partitioned Postgres or a purpose-built ledger like TigerBeetle; hot accounts like fee revenue can be split into sub-accounts and summed.

Even with perfect code, external parties make mistakes, files arrive late, and some failures leave no trace in our system. Reconciliation compares three views daily: our payments and ledger, the processor's settlement report, and the bank statement. Matching keys are our processor_ref, amount and date. Typical exceptions: captured at the processor but failed in our system (timeout we resolved wrongly), fee different from contract, settlement missing or short, duplicate settlement. Each exception creates a case and, after review, a correcting ledger journal. Track the match rate and age of open exceptions as key metrics. Reconciliation is also what makes the 'unknown' state safe in the end: anything not resolved online is resolved here.

Payments need transactions across the payment row, the idempotency key and the outbox, plus unique constraints. Postgres (or a distributed SQL like Spanner/CockroachDB) gives that directly. NoSQL can scale writes more easily but you rebuild transactions yourself. Shard Postgres by merchant_id to scale.

Posting ledger entries in the same transaction as the payment is simplest when they share a database. At scale the ledger is its own service; posting asynchronously from the outbox keeps payments fast, and integrity checks ensure every payment event gets exactly one journal.

Separate capture suits marketplaces and travel (capture when shipped or ticketed) and makes cancellations free (void instead of refund). Auto-capture is simpler for digital goods. Support both; track authorisation expiry.

An in-house vault reduces dependence and fees but brings full PCI DSS Level 1 scope. A third-party vault speeds launch. Either way, keep raw card data out of the main services.

One merchant drives most traffic, so one shard and one merchant balance account get hot. Split the merchant's ledger account into N sub-accounts and sum them for balance; payments can be spread by payment_id within the merchant's shard group.

A processor slows down and threads pile up. Use async I/O, per-processor bulkheads and circuit breakers, and route to a secondary processor when the primary degrades (where the merchant contract allows).

A merchant endpoint is slow or down, and retries pile up. Queue per merchant with exponential backoff, cap concurrency per endpoint, and let merchants fetch events via API.

The payment stays in processing. The recovery worker finds stale processing payments, queries the processor by our reference, and completes the transition. The merchant's retry with the same Idempotency-Key returns the in-progress or final result.

Payments still commit to Postgres with outbox rows. The relay catches up when Kafka returns; ledger and webhooks are delayed but not lost.

A continuous check finds a journal or account that does not balance. Freeze payouts for affected accounts, page on-call, and fix with a correcting journal after root cause; never edit history.