Kafka: partitions, consumer groups and ordering
In one line
Kafka is a replicated, partitioned, append-only log: producers append records to partitions chosen by key, consumer groups split partitions among their members and track offsets, and ordering is guaranteed only within one partition.
What it is
A Kafka topic is split into partitions. Each partition is an ordered, append-only log stored on disk; every record gets a sequential offset. Records are not deleted when read: they are kept for a retention period (for example 7 days) or until a size limit, or kept as the latest value per key in a compacted topic. This is the big difference from a classic queue such as RabbitMQ or SQS, where a message is removed once acknowledged.
Each partition has one leader broker and some followers (replication factor, usually 3). Producers write to the leader; followers copy. The in-sync replicas (ISR) are the followers that are caught up. With acks=all and min.insync.replicas=2, a write is confirmed only when at least two replicas have it, so losing one broker loses no acknowledged data.
The producer picks a partition by key: by default hash(key) mod number_of_partitions (murmur2). All records with the same key go to the same partition and stay in order. Records without a key are spread across partitions (the 'sticky' partitioner fills a batch for one partition, then moves on).
Consumers join a consumer group. Kafka assigns each partition to exactly one consumer in the group, so the group's parallelism is capped by the partition count. Each group commits its own offsets (stored in the internal __consumer_offsets topic), so many independent groups can read the same topic at their own pace: search indexing, analytics and notifications each get every event. Since Kafka 4.0, clusters run on KRaft (Kafka's own Raft-based metadata quorum) instead of ZooKeeper.
Why interviewers ask
Kafka appears in almost every large design: event-driven microservices, CDC pipelines, activity feeds, log and metrics ingestion, ad click aggregation. Interviewers check that you understand partitions as the unit of ordering and parallelism, how to choose a partition key, what consumer lag means, how many partitions to create, what rebalancing does, and the durability settings. Saying 'we will use Kafka' without a key, a partition count and an ordering argument is a common way to lose points.
How it works
- 1Create a topic with a partition count and replication factor, for example orders.events with 48 partitions and RF=3 across 3 availability zones.
- 2Producers batch records per partition (linger.ms, batch.size), compress them (lz4 or zstd) and send them to partition leaders. Batching is why Kafka reaches high throughput.
- 3Durability settings: acks=all, min.insync.replicas=2 and enable.idempotence=true (the default since Kafka 3.0). The idempotent producer attaches a producer id and sequence numbers, so broker-side retries do not create duplicates or reorder records within a partition.
- 4Choose the key by what needs ordering: order_id for an order's lifecycle, user_id for a user's activity, account_id for a ledger. Events for one key are processed in order; events for different keys are processed in parallel.
- 5Consumers in a group poll records, process them and commit offsets. Commit after processing to get at-least-once delivery; committing before processing risks losing records on a crash.
- 6When consumers join, leave or crash (missed heartbeats or max.poll.interval.ms exceeded), the group rebalances and partitions are reassigned. Cooperative sticky assignment and static membership reduce the pause.
- 7Monitor consumer lag (latest offset minus committed offset) per partition. Growing lag means consumers are too slow; scale up to the partition count, speed up processing, or fix a hot partition.
- 8Handle poison messages with retry topics and a dead-letter topic, so one bad record does not block its partition forever.
- 9Plan partition count up front: adding partitions later changes hash(key) mod N, so new records for an existing key may go to a different partition than old ones and per-key ordering breaks across the change.
Worked example · Order events for a food delivery platform
- Load: 2M orders per day, each with about 10 lifecycle events (placed, accepted, preparing, picked up, delivered...) = 20M events per day, about 230 per second on average and maybe 2,000 per second at dinner peak. Records are about 1 KB, so peak is about 2 MB per second, small for Kafka.
- Topic: orders.events, key = order_id, 24 partitions, RF=3, retention 7 days. Partitions are sized for consumer parallelism, not broker throughput: the slowest consumer (a fraud scorer that calls a model, 50 ms per event) needs 2,000 x 0.05 = 100 events in parallel, so it processes in small concurrent batches per partition.
- Ordering: all events for order 981 land on one partition, so the tracking service never sees 'delivered' before 'picked up' for the same order.
- Consumer groups: tracking-service, notification-service, analytics-loader and fraud-scorer each read every event independently with their own offsets.
- A consumer crashes after processing offset 5,120 but before committing; on restart it reads from the last committed offset 5,100 and reprocesses 20 events. Consumers deduplicate by event_id, so customers do not get duplicate notifications.
- A restaurant chain with a single 'mega-kitchen' key would be a hot partition; keying by order_id instead of restaurant_id avoids it, because ordering is needed per order, not per restaurant.
Picture it
A topic is like a set of numbered checkout lanes in a supermarket. Each customer (key) always uses the same lane, so their items are scanned in order. More lanes means more cashiers can work at once, but there is no single order across all lanes. A consumer group is a team of cashiers; one lane cannot have two cashiers at the same time.
Key patterns
Key by the entity that needs order
When: Every topic
How: Use order_id, account_id or user_id. Narrower keys spread load better; broader keys give wider ordering but risk hot partitions.
Over-provision partitions once
When: Creating a topic
How: Pick a count that covers expected peak consumer parallelism for a couple of years (for example 12-64 for most services). Avoid thousands per topic; each partition costs broker memory, file handles and recovery time.
Retry topics and dead-letter topic
When: Consumers that can fail on some records
How: On failure, publish to orders.retry.1m, then orders.retry.10m, then orders.dlq with error details; keep the main partition moving and replay the DLQ after a fix.
Compacted topics for latest state
When: You need the current value per key (prices, user settings, CDC tables)
How: cleanup.policy=compact keeps at least the latest record per key; a null value (tombstone) deletes the key. New consumers rebuild state by reading the topic.
Real-world use
Kafka was created at LinkedIn for activity tracking and log data. LinkedIn has reported processing trillions of messages per day across its Kafka clusters.
Uber
Uber uses Kafka as the backbone for trip events, driver locations and logging, and built uReplicator for cross-region replication and consumer proxies to deliver to services at large scale.
Netflix
Netflix's Keystone pipeline routes trillions of events per day through Kafka into stream processors and data stores for analytics and operational use.
Indian product companies
Companies like Swiggy, Flipkart and Zomato run event-driven architectures where order, delivery and catalogue events flow through Kafka to many downstream services; their engineering blogs describe Kafka-based pipelines for tracking, analytics and data ingestion.
Managed Kafka
Confluent Cloud, Amazon MSK and Aiven run Kafka as a service; Redpanda and WarpStream offer Kafka-compatible APIs with different storage designs, such as object-storage-backed brokers.
In a project
Your payments team publishes payment status events keyed by merchant_id. One large merchant creates 40% of all events during a sale, its partition's lag reaches 30 minutes, and merchants see stale payment statuses.
- Confirm the hot partition with per-partition lag and bytes-in metrics; the other 47 partitions are nearly idle.
- Re-examine the ordering requirement: order matters per payment, not per merchant. Change the key to payment_id, which spreads the big merchant across all partitions.
- Because changing the key changes partition placement, deploy at a quiet time and have consumers tolerate a short window of reordering by checking a version or status-transition rule per payment.
- Scale the consumer group to 48 instances (one per partition) and process records concurrently within a partition by key, using a library such as Confluent's parallel consumer, keeping per-key order.
- Add alerts on per-partition lag, not just total lag, so a single hot partition is visible.
- Document the rule: pick the narrowest key that still gives the ordering you need.
Impact: Peak lag drops from 30 minutes to under 5 seconds, statuses are fresh during sales, and the team stops trying to fix the problem by adding consumers that could never help a single partition.
Trade-offs
Use it when
- Many independent consumers need the same events (fan-out to several services)
- You need replay: rebuild a search index or a projection from past events
- High-throughput streams: clickstream, logs, metrics, location pings, CDC
- Per-entity ordering with horizontal scale
Avoid it when
- Simple task queues with per-message acknowledgement, delays and priorities; SQS, RabbitMQ or a database-backed queue is simpler
- You need a single global order across all events; that means one partition and limited throughput
- Very small teams without operational capacity, unless you use a managed service
- Request-response calls that need an immediate answer
Common mistakes
- Claiming Kafka guarantees order across a topic; it is per partition only.
- Adding more consumers than partitions and expecting more throughput.
- Increasing partition count on a keyed topic without realising it changes key-to-partition mapping.
- Using a low-cardinality or skewed key (country, merchant, a celebrity user) and creating a hot partition.
- Committing offsets before processing (data loss on crash) or using acks=1 for financial events.
- Slow processing inside the poll loop exceeding max.poll.interval.ms, causing endless rebalances.
- Ignoring consumer lag and retention: if lag exceeds retention, unread data is deleted.
Interview questions
Quick revision
- Topic = partitions; partition = ordered append-only log with offsets.
- Order is per partition only. Key by the entity that needs order.
- Default partition = murmur2(key) mod partitions; adding partitions remaps keys.
- Group parallelism <= partition count.
- Each group has its own offsets; many groups read the same data.
- Durability: RF=3, acks=all, min.insync.replicas=2, idempotent producer.
- Commit after processing = at-least-once; make consumers idempotent.
- Watch per-partition lag; lag beyond retention = data lost to that consumer.
- Retention by time/size, or compaction to keep the latest value per key.
- KRaft replaces ZooKeeper (Kafka 4.0 removes ZooKeeper).
Check yourself
1. A topic has 16 partitions. What is the maximum number of consumers in one group that can actively read from it?
2. Events for the same order must be processed in order. What should the producer do?
3. Which settings give strong durability for payment events?
4. You increase a keyed topic from 12 to 24 partitions. What is the risk?
See it in a full design
These walkthroughs use this lesson.
Saved on this device. Create a free account to keep it on every device.
Practise this with Sam
A real mock interview, out loud, with an honest scorecard. Free to start.