Concurrency in practice

Concurrency means structuring a program so several tasks make progress in overlapping time; parallelism means they actually run at the same instant on different cores. A web server handling 10,000 open connections is concurrent; a video encoder using 8 cores is parallel.

Instead of creating raw threads and locks everywhere, production code uses a small set of patterns: thread pools (a fixed set of reusable workers), producer-consumer with a bounded queue, event loops with async I/O (one thread juggling many waiting tasks), process pools for CPU-heavy work, and message passing (channels or actors) instead of shared memory.

1. 1Classify the work. I/O-bound tasks (HTTP calls, database queries, file reads) mostly wait; CPU-bound tasks (image resizing, compression, ML inference) mostly compute.
2. 2For I/O-bound work, use a thread pool or an event loop. While one task waits on the network, others run. One event loop thread can handle thousands of idle connections because waiting costs no CPU.
3. 3For CPU-bound work, use about one worker per core (a process pool in Python because of the GIL, threads in Java or Go). More workers than cores only adds switching.
4. 4Connect stages with a bounded queue (producer-consumer). Producers add jobs; consumers take them. When the queue is full, producers block or are rejected, which is backpressure.
5. 5Prefer passing messages over sharing state. If state must be shared, keep it small and protected, or use thread-safe structures.
6. 6Plan shutdown and failures: timeouts on every external call, sentinel values or cancellation to stop workers, and retries for idempotent operations.

import queue
import threading

tasks = queue.Queue(maxsize=100)  # bounded: put() blocks when full (backpressure)
STOP = None                       # sentinel that tells a worker to exit

def worker(name):
    while True:
        item = tasks.get()        # blocks until an item is available
        if item is STOP:
            break
        print(f"{name} resized image {item}")

workers = [threading.Thread(target=worker, args=(f"w{i}",)) for i in range(4)]
for w in workers:
    w.start()

for image_id in range(20):        # the producer
    tasks.put(image_id)
for _ in workers:                 # one sentinel per worker
    tasks.put(STOP)
for w in workers:
    w.join()

queue.Queue is thread-safe, so no explicit lock is needed. The maximum size of 100 makes put() block when consumers fall behind. Each worker exits when it reads the sentinel, and putting one sentinel per worker guarantees all four stop.

Cost: A common rule of thumb for pool size is cores x (1 + wait time / compute time): about the core count for CPU-bound work, many more for I/O-bound work, capped by downstream limits like database connections.