Design shipment tracking for a logistics company (Delhivery)

Hubs and delivery vans lose connectivity all the time, so the scanner must never block an operator on the network. Each scan is stored locally with a UUID generated on the device and the device timestamp, and uploaded in batches with retries. This guarantees at-least-once delivery, so the server must be idempotent: dedupe on event_id (Redis set with TTL for speed, and the event's primary key in Cassandra as a natural guard). Events also arrive out of order: a device offline for two hours uploads an 'inbound at Bhiwandi hub' scan after the parcel was already scanned outbound at the next hub. Store every event in the history, sorted by event time, but compute status with rules: a newer status wins over an older one, terminal states (DELIVERED, RTO_DELIVERED) are never regressed by older events, and some transitions are flagged as anomalies (delivered then in-transit) for ops review instead of being applied. Device clocks drift, so record received_at too, and correct obviously wrong device times (in the future, or before pickup) using server time. Kafka partitioning by AWB keeps processing of one shipment serialised.

Treat scan events as the source of truth and the current status as a derived view, which is a light form of event sourcing. The projector is the only writer of shipment_status. For each event, it loads the status row, applies a pure function (current_state, event) → new_state, and writes with a conditional update on version, so a concurrent writer or a retried message cannot apply the same change twice. Because the function is pure, you can replay a shipment's events from Cassandra to rebuild its status after a bug fix, or replay a whole day from S3 into a new projection such as 'hub dwell time'. The projection also holds what tracking needs in one read: current location, next planned hub from the manifest route, attempts count and EDD. Keep the event schema versioned and additive. A common mistake is letting many services update status directly (ops tools, delivery app, returns system); then nobody can explain why a parcel shows 'delivered' in one screen and 'in transit' in another. Ops corrections should also be events (MANUAL_STATUS_OVERRIDE with a reason) so the audit trail stays complete.

Merchants such as marketplaces and D2C brands rely on webhooks to update their own order pages, so delivery reliability matters more than speed. Build it as an outgoing webhook service: consume StatusChanged events, render a signed payload (HMAC with the merchant's secret) carrying event_id, awb and status, and POST it with a timeout of a few seconds. Keep a separate queue or rate limit per merchant so one merchant whose endpoint is down does not consume all workers; retry with exponential backoff and jitter up to 24 hours, then park in a dead-letter queue with a replay button in the merchant dashboard. Merchants must treat event_id as an idempotency key because retries can duplicate. Order is not guaranteed across retries, so include status_ts and let merchants ignore older updates. Customer messages cost money (SMS, WhatsApp template messages), so send only milestones and collapse rapid changes: if 'out for delivery' and 'delivered' happen within 2 minutes, skip the first. Use regional language templates based on the destination state, which noticeably reduces failed deliveries and support calls.

The promise shown at checkout and on the tracking page is a prediction, not a lookup. Start from the planned route (origin hub → line-haul → destination hub → last-mile centre) and the historical transit time distribution for each lane and hub, by day of week and season. A model (gradient-boosted trees work well) predicts arrival at each remaining step using features such as current hub backlog, cut-off times for the next truck, holidays, festive load and weather. Recompute on every status change, and also on a timer for shipments that have not moved: a parcel that missed its 22:00 line-haul cut-off is late before any new scan happens. Show a date, not a timestamp, and only move it later once with a reason ('delayed due to heavy rain in Assam'), because flip-flopping erodes trust. Feed predicted delays to ops as exceptions so they can act (re-route, prioritise) before the customer notices. Track calibration: the share of shipments delivered on or before the promised date by lane.

Postgres could store ~60M rows/day with partitioning, and is simpler for small companies. Cassandra fits better at this volume: write-heavy, append-only, read by one key (AWB), easy horizontal scaling and TTL. Keep the status projection in a store with conditional writes (DynamoDB or sharded Postgres), and do analytics in a warehouse, not on either.

Polling the tracking API from thousands of merchants every few minutes multiplies read load and still feels slow. Webhooks push changes once. Offer both: webhooks for real-time and a bulk status API with since-cursor for reconciliation, because webhooks will sometimes be missed.

You cannot guarantee arrival order from offline devices, so do not try to enforce global order. Partition by AWB for per-shipment serial processing, and make the state function tolerant of late events. This is cheaper and more robust than buffering events and waiting for missing ones.

Thousands of devices sync at shift end, and a sorter machine can emit hundreds of scans per second. Kafka absorbs bursts; ingestion is stateless and autoscaled; projector consumers scale up to the partition count (plan 128+ partitions). Watch consumer lag as the key SLI: lag directly means stale tracking.

During sales, customers refresh tracking pages repeatedly. CDN caching for 30-60 s and Redis caching per AWB make reads cheap; the origin sees only misses. Collapse concurrent misses for the same AWB (request coalescing).

Ingest returns errors and devices keep events in their local queue, retrying later. No scan is lost because the device deletes events only after ack. Tracking shows slightly stale status; alert on device queue depth reported in heartbeats.

Because events are the source of truth, fix the code and replay affected AWBs (or a time window) from Cassandra or S3 to rebuild the projection. Suppress notifications during replays so customers do not get duplicate messages.

Its per-merchant queue backs off and grows while others are unaffected. After 24 h events move to a dead-letter queue; the merchant can replay them or use the bulk status API to reconcile.