Design credit card bill payments and rewards (CRED)

The core risk is money stuck between the two legs: collected from the user but not yet paid to the issuer. Model each payment as an explicit state machine: CREATED → COLLECTING → COLLECTED → PAYING_ISSUER → PAID, with side branches COLLECT_FAILED, ISSUER_UNKNOWN, REFUND_PENDING and REFUNDED. Store the state in Postgres and move it only with conditional updates (UPDATE payments SET state = 'PAYING_ISSUER', version = version + 1 WHERE id = ? AND state = 'COLLECTED' AND version = ?), so a webhook, a poller and a retry can all race safely. Run the multi-step process in a durable workflow engine such as Temporal, or a home-grown 'job table + workers' if you want fewer moving parts: each step is an activity with retries, timeouts and an idempotency key, and the workflow resumes after crashes. Every external call carries a reference that the partner deduplicates on (the PG order id, the BBPS reference = payment_id). Treat timeouts as UNKNOWN and resolve them by status queries, never by blind retry with a new reference, or the issuer may receive the money twice. Finally, set SLAs per state: a payment in COLLECTED for more than 10 minutes pages on-call. This is a saga: the compensating action for a failed issuer leg is a refund, not a rollback.

Webhooks get lost and partners make mistakes, so the database is not trusted until reconciled. Every day, and intraday for faster detection, download three sources: the payment gateway settlement file (money collected and fees), the BBPS / BBPOU settlement report (bill payments made to issuers), and your own payments table. Load them into staging tables and match on references: pg_txn_id and payment_id, bbps_txn_ref. Each payment should land in exactly one bucket: matched (collected and paid), collected-not-paid, paid-not-collected (rare, means a bug), or refunded. Breaks create tickets and automatic actions: collected-not-paid older than the SLA triggers a status query and then a refund; amount mismatches go to finance ops. The nodal or escrow account balance must also match the sum of money in flight. Keep the raw files in S3 for audit. In an interview, saying 'I reconcile three ledgers daily and alert on breaks, because webhooks are a hint and settlement files are the truth' is a strong signal. Also note the user-facing angle: when a payment is PAID on BBPS, the issuer may take a little time to reflect it on the card, so show 'paid to bank, will reflect on your card soon' instead of confusing users.

Rewards are money-like: users notice missing coins, and finance notices overspent budgets. Use an append-only ledger with a balance row, updated in one transaction, never a single mutable 'coins' column. Credit only on the final PAID state, through the outbox event, so a payment that is later refunded does not keep its reward; if it does get reversed, write a 'reverse' entry. Idempotency comes from a unique constraint on (type, source_ref): replaying Kafka from the start credits nothing twice. Campaign rules ('2x coins on bills above ₹10,000 paid 3+ days before due date, max 5,000 coins per user per month') live in a rules engine with versioned rule sets, so you can explain any credit later. Budgets are the tricky part at peak: checking a Postgres row per credit makes it a hot row. Instead, keep the remaining budget in Redis and use an atomic DECRBY (or a Lua script that checks per-user caps too); on success, write the ledger. If Redis and the ledger disagree after a crash, a reconcile job fixes Redis from the ledger. Accept a tiny overspend window and size alerts to it. Add abuse checks: many cards on one device, repeated small payments to farm coins, and payments refunded right after the reward.

Bills come from BBPS bill fetch, which calls the issuer through the BBPS network. These calls are slow (seconds), sometimes fail, and may be rate limited by issuers. So never fetch on every app open. Cache the latest bill per card with a TTL of a few hours, and refresh in the background around the predicted statement date (learned from history: statements usually arrive on the same day each month). Spread refresh jobs with jitter so ten million cards with a statement on the 5th are not fetched at 00:00. Use per-issuer concurrency limits and circuit breakers: if one issuer's biller is down, mark its bills as 'last updated 2 days ago' instead of failing the screen. Reminders are scheduled jobs: when a bill is fetched, create reminder tasks at due_date - 5 days and due_date - 1 day, cancelled when the bill is paid. A time-bucketed scheduler (tasks partitioned by hour) handles millions of tasks a day. Respect quiet hours and send via push first, SMS only for important reminders, because SMS costs money and has DLT template rules in India.

Temporal gives retries, timers and resumable steps for free and makes the flow readable. It adds an operational dependency. A Postgres state machine plus a polling worker is simpler and is enough at ~1,000 payments/s. Either way, keep the payments table as the source of truth for state that users and finance see.

Waiting for the issuer can take seconds to minutes. Showing success right after collection risks a confusing reversal later. The usual answer: show 'payment received, sending to your bank' after collection, and 'paid' after BBPS confirms, with a push notification. Honest intermediate states reduce support tickets.

Crediting instantly feels great but is exposed to refunds and fraud. Crediting after settlement is safe but feels slow. Credit on PAID, hold redemption of high-value rewards for a short cooling period, and reverse on refund.

At 10x volume, BBPS and PG partners become the limit, not your servers. Use per-partner rate limiters and queue payments into PAYING_ISSUER at the rate partners accept; keep the user-facing part fast. Use multiple PG and BBPOU partners with health-based routing.

A popular campaign means thousands of credits per second on one budget. Shard the budget into N Redis counters (budget split across sub-keys) and pick one at random; when one runs out, try another. This removes the hot key at the cost of slightly less exact limits.

The payment stays in COLLECTING. A poller queries the PG status API for payments older than a few minutes, and reconciliation catches anything left. Webhooks speed things up but are never the only path.

Payments queue in COLLECTED and retry with backoff using the same reference. Users see 'delayed at bank'. If the outage passes the SLA (and the due date is close), the user is offered a refund. Before collecting new money, check the issuer's health and warn users or block payment for a down issuer, so money is not collected that cannot be delivered.

Payments are unaffected because rewards consume events asynchronously. Kafka retains the events; when the service recovers it catches up, and the ledger's unique constraint keeps it idempotent.