Message queues & async processing
In one line
A message queue lets one part of a system hand off work as messages that other parts process later, decoupling services, absorbing traffic spikes and keeping user-facing requests fast.
What it is
A message queue (or broker) is a durable buffer between producers that send messages and consumers that process them. The producer does not wait for the work to be done; it only waits for the broker to store the message.
There are two main styles. Work queues (RabbitMQ, Amazon SQS) hand each message to one consumer and delete it once acknowledged. Logs or streams (Apache Kafka, Amazon Kinesis) append messages to partitioned, ordered logs that are kept for a retention period, and each consumer group tracks its own position (offset), so many independent services can read the same events.
Why interviewers ask
Most large designs move slow or fan-out work off the request path: sending emails, processing uploads, updating feeds, notifying other services. Interviewers check that you know when async helps, what delivery guarantees really mean, and how to handle duplicates, ordering, retries and poison messages.
How it works
- 1The producer publishes a message (for example {order_id: 981, event: 'placed'}) to a queue or topic and returns to the user as soon as the broker confirms it is stored.
- 2The broker persists the message, often replicated across several nodes so a broker failure does not lose it.
- 3Consumers pull (or are pushed) messages, do the work, then acknowledge. Unacknowledged messages are redelivered after a visibility timeout or on consumer restart.
- 4Because a consumer can crash after doing the work but before acknowledging, most systems give at-least-once delivery, so consumers must be idempotent (processing the same message twice has the same effect as once).
- 5Scale consumers horizontally. In Kafka, a topic's partitions are divided among the consumers in a group; order is guaranteed only within a partition, so key messages by the entity whose order matters (for example order_id).
- 6Messages that keep failing go to a dead-letter queue after a few retries with backoff, so one bad message does not block the rest.
- 7Watch the queue depth or consumer lag: if it keeps growing, consumers cannot keep up and you need more of them.
Worked example · Order placed: sync vs async
- Synchronous: the checkout API saves the order (30 ms), charges the card (300 ms), sends the email (400 ms) and updates analytics (100 ms). The user waits about 830 ms, and an email provider outage fails the whole checkout.
- Asynchronous: the API saves the order and charges the card (330 ms), then publishes OrderPlaced to Kafka (5 ms) and returns. The user waits about 335 ms.
- The email service and the analytics service each have their own consumer group, so both read the same OrderPlaced event independently.
- The email provider is down for 10 minutes: the email consumer retries with backoff, its lag grows to 6,000 messages, then drains when the provider recovers. Checkout never noticed.
- The email consumer crashes after sending the email for order 981 but before committing its offset. On restart it reads order 981 again, checks a sent_emails table keyed by order_id, sees it was sent, and skips it. Idempotency prevents a duplicate email.
Picture it
A restaurant's order rail: waiters clip tickets to the rail and go back to customers, and cooks take tickets when they are free. A rush makes the rail longer, not the waiters slower, and more cooks can work through it in parallel.
Key patterns
Work queue (competing consumers)
When: Background jobs where each task should be done once
How: Many workers pull from one queue; each message goes to one worker and is deleted on ack.
Publish-subscribe / event log
When: Several services react to the same event
How: Publish to a topic; each service has its own subscription or consumer group with its own position.
Idempotent consumer
When: Always, given at-least-once delivery
How: Use a unique message or business id and record processed ids, or design writes as upserts.
Dead-letter queue with backoff
When: Messages that fail repeatedly
How: Retry with increasing delays, then move to a DLQ for inspection instead of blocking the queue.
Transactional outbox
When: You must update the database and publish an event reliably
How: Write the event to an outbox table in the same DB transaction, and a relay publishes it to the broker.
Real-world use
LinkedIn and Apache Kafka
Kafka was created at LinkedIn to move activity events and logs between systems, and it is now used by companies like Uber and Netflix to stream trillions of events a day between services.
Amazon SQS
A managed work queue with visibility timeouts and dead-letter queues; standard queues give at-least-once delivery and best-effort ordering, while FIFO queues give ordering and deduplication within a message group.
Background jobs (Sidekiq, Celery)
Rails and Python apps push jobs such as emails, image resizing and report generation onto Redis or RabbitMQ and process them with worker pools.
Video processing pipelines
Upload services like YouTube put new videos into a processing pipeline so transcoding into many resolutions happens in the background while the upload request returns quickly.
In a project
A photo-sharing app's upload endpoint takes 4 seconds because it resizes images into 5 sizes, runs moderation and updates followers' feeds inline; it times out during peaks.
- Store the original in object storage, write a photo row with status 'processing', publish PhotoUploaded to a queue and return in about 200 ms.
- Run separate consumers for resizing, moderation and feed fan-out, each scaled on its own queue depth.
- Make every consumer idempotent: resizing writes to deterministic keys (photo_id/size), feed fan-out uses upserts keyed by (user, photo).
- Retry failures with exponential backoff, send poison messages to a dead-letter queue and alert on it.
- Mark the photo 'ready' when all required steps are done, and show a placeholder until then.
Impact: Upload latency drops from 4 s to 200 ms, timeouts disappear, a moderation-service outage delays processing instead of failing uploads, and worker fleets scale independently to absorb 10x upload spikes.
Trade-offs
Use it when
- Work the user does not need to wait for (emails, thumbnails, notifications)
- Traffic spikes that would overwhelm a downstream service
- Several services need to react to the same event without the producer knowing about them
Avoid it when
- The caller needs the result right now to respond (validate a password, check stock before confirming)
- Simple systems where a direct call is clearer and the extra broker is not worth operating
- You need strict global ordering across all messages at high throughput
Common mistakes
- Claiming 'exactly-once delivery' casually; in practice design for at-least-once plus idempotent consumers.
- Assuming global ordering in Kafka; order is only guaranteed within a partition.
- Publishing the event and writing the database in two separate steps, so a crash between them loses or invents events.
- No dead-letter queue, so one malformed message blocks a consumer forever.
- Not monitoring consumer lag, so a slow consumer silently falls hours behind.
Interview questions
Quick revision
- Queue = durable buffer; producer returns once the broker has stored the message.
- Work queue: each message to one consumer. Pub-sub/log: every subscriber group gets every message.
- Assume at-least-once; make consumers idempotent.
- Kafka orders messages per partition only; key by the entity that needs order.
- Retries with backoff, then dead-letter queue.
- Monitor queue depth and consumer lag.
- Outbox pattern for 'write DB and publish event' atomically.
Check yourself
1. A consumer processes a message, then crashes before acknowledging it. With at-least-once delivery, what happens?
2. In Kafka, which statement about ordering is true?
3. What is a dead-letter queue for?
4. Which task is the worst fit for asynchronous processing via a queue?
See it in a full design
These walkthroughs use this lesson.
Saved on this device. Create a free account to keep it on every device.
Practise this with Sam
A real mock interview, out loud, with an honest scorecard. Free to start.