Streaming with Apache Kafka

What you will build: a one-machine Kafka setup in Docker, a topic for Brew & Bean order events, a Python producer that publishes status changes, and a Python consumer in a consumer group that reads them. You will see partitions, keys, offsets and consumer groups working.

Prerequisites: Python, the command line and Docker Desktop (or Docker Engine) installed. No Java install is needed because Kafka runs inside the container. Read *ETL vs ELT and pipeline design* first if batch versus streaming is new to you.

So far the pipelines in this track have been batch: run once an hour or day on a pile of data. Many things cannot wait: fraud checks, live order screens in a kitchen, delivery tracking, live dashboards. They need to react to events (an order was placed, a payment failed) within seconds.

Apache Kafka is a distributed event log. Programs called producers append events to it. Programs called consumers read them, each at their own pace. Kafka stores events on disk for a configurable time, so a slow or crashed consumer can catch up and a new consumer can replay history. It decouples systems: the ordering app does not need to know that the kitchen screen, the analytics pipeline and the SMS service all want its events. It just publishes them.

Think of a notice board with a numbered sequence of notes. Anyone can pin a note at the end. Each reader keeps their own bookmark and reads forward from it. Notes stay on the board for a while. Nobody removes a note because someone read it.

Two rules explain most Kafka behaviour.

Order is guaranteed only within a partition. Events for different partitions have no global order. That is why we use the order id as the key: all events for order 101 (received, preparing, ready) go to the same partition and are read in that order.

Within a consumer group, each partition is read by exactly one consumer. If the orders topic has 3 partitions and a group has 3 consumers, each reads one partition, so you scale reading by adding consumers (up to the number of partitions). A different group, say analytics, independently reads all the events again. A second consumer beyond the partition count would sit idle.

Kafka has two other pieces you will see later: replication (each partition is copied to other brokers so a broker can fail without data loss) and Kafka Connect (ready-made connectors that copy data between Kafka and databases, S3 and so on). Older Kafka versions needed a separate ZooKeeper service; current versions use a built-in mechanism called KRaft, which is what the Docker image uses.

docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 3 --replication-factor 1

docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic orders

Creates the orders topic with 3 partitions (replication factor 1 because we have one broker) and shows its layout. Lines ending with a backslash continue the command; in PowerShell put the whole command on one line.

Open two terminals. In the first, start a consumer that reads from the beginning of the topic. In the second, start a producer and type some lines.

# Terminal 1: consumer
docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 --topic orders --from-beginning

# Terminal 2: producer (type a line and press Enter; Ctrl+C to stop)
docker exec -it kafka /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 --topic orders

Anything you type into the producer appears in the consumer within a moment. Stop both with Ctrl+C.

This works, but the messages have no keys and are just text. In real systems your own programs produce and consume. Let us write them in Python.

import json
import time

from confluent_kafka import Producer

producer = Producer({"bootstrap.servers": "localhost:9092"})


def report(err, msg):
    """Called once Kafka confirms (or rejects) each message."""
    if err is not None:
        print(f"FAILED: {err}")
    else:
        print(f"sent to partition {msg.partition()} at offset {msg.offset()}")


events = [
    {"order_id": 101, "status": "received", "item_id": "cappuccino", "qty": 2},
    {"order_id": 102, "status": "received", "item_id": "masala-chai", "qty": 1},
    {"order_id": 101, "status": "preparing"},
    {"order_id": 101, "status": "ready"},
    {"order_id": 102, "status": "preparing"},
    {"order_id": 102, "status": "ready"},
]

for event in events:
    producer.produce(
        "orders",
        key=str(event["order_id"]),          # same key -> same partition -> in order
        value=json.dumps(event),
        callback=report,
    )
    producer.poll(0)                          # let delivery callbacks run
    time.sleep(0.2)

producer.flush()                              # wait until everything is delivered

Publishes six order status events. The key is the order id, so events of one order always land in the same partition and keep their order. produce() is asynchronous: it queues the message, and the callback tells you when Kafka has accepted it.

Notice what the pattern shows: all three events for order 101 share a partition and have increasing offsets, so a consumer sees received, then preparing, then ready. producer.flush() at the end matters: without it the program may exit before queued messages are sent.

import json

from confluent_kafka import Consumer

consumer = Consumer({
    "bootstrap.servers": "localhost:9092",
    "group.id": "kitchen",
    "auto.offset.reset": "earliest",   # a new group starts from the beginning
    "enable.auto.commit": False,       # we commit only after processing
})
consumer.subscribe(["orders"])

try:
    while True:
        msg = consumer.poll(1.0)       # wait up to 1 second for a message
        if msg is None:
            continue
        if msg.error():
            print(f"Consumer error: {msg.error()}")
            continue

        event = json.loads(msg.value())
        print(f"partition={msg.partition()} offset={msg.offset()} "
              f"key={msg.key().decode()} -> order {event['order_id']} is {event['status']}")

        consumer.commit(message=msg)   # record progress: this message is done
except KeyboardInterrupt:
    pass
finally:
    consumer.close()                   # leave the group cleanly

Reads the orders topic as part of the group kitchen. Stop it with Ctrl+C.

Why does the consumer commit manually? Offsets record how far a group has read. If the consumer crashes and restarts, it resumes from the last committed offset. If you commit before processing and then crash, that message is lost (at-most-once). If you commit after processing and crash before committing, the message is processed again after restart (at-least-once). At-least-once is the usual choice, which means your processing must tolerate duplicates: make it idempotent, for example by updating a row by order_id rather than inserting a new one. This is the same idempotency idea as in the Airflow lesson. Kafka also offers exactly-once processing in some setups (transactions between Kafka topics), but it needs careful configuration, so design for at-least-once first.

docker stop kafka
docker rm kafka

Stops and removes the container and its data.

In real pipelines Kafka sits between producers and many consumers: an app writes order events, Kafka Connect or a stream processor copies them to a data lake or warehouse, and a processing engine computes live aggregates. The next lesson, *Stream processing: Flink and Spark Structured Streaming*, covers that last part.

Managed Kafka services exist on every major cloud, plus Confluent's platform; alternatives such as Redpanda speak the Kafka protocol too. Managed options save you from running brokers yourself.

How this comes up in interviews: the difference between a topic, partition and offset; why order is guaranteed only within a partition; how consumer groups scale reads; at-most-once, at-least-once and exactly-once; what consumer lag is; how to choose a key and the number of partitions (more partitions allow more parallelism but are hard to reduce later, and a skewed key makes one partition a hot spot); and how Kafka differs from a queue like RabbitMQ (Kafka retains events and lets many groups replay them).

You are a streaming data engineer. I am designing Kafka topics for [describe your system, e.g. a food delivery app with orders, payments and rider locations]. Suggest topic names, the key for each, a sensible partition count with reasoning, the retention period, and the consumer groups that would read each topic. Point out one risk such as hot partitions or ordering problems.

What to expect: A good answer picks keys that keep related events together (for example order_id), explains that ordering only holds per key, and warns about skew. Treat partition counts as estimates and verify them against your real throughput.