Background jobs and queues

When a customer places an order, your API must save it, and also send an SMS receipt, notify the kitchen and maybe email an invoice. If the request waits for all of that, the customer stares at a spinner for several seconds, and if the SMS provider is down the whole order fails. The fix is to do only the essential work during the request and hand the rest to a background job.

What you'll learn and build: first FastAPI's built-in BackgroundTasks for small jobs, then a proper job queue with a separate worker process using Redis and the arq library: enqueue a receipt job from the API, run it in a worker, retry it when it fails, and check its status.

Prerequisites: Python 3.10+, basic FastAPI, Redis running locally (see *Caching with Redis* for the Docker command) and curl.

Background jobs give you three things: speed (the response is fast), reliability (a failed job can be retried without failing the customer's request) and smoothing (a sudden burst of orders becomes a queue that workers drain at their own pace).

For small, quick, non-critical work, FastAPI can run a function after it has sent the response, in the same process. Save this as main.py, run uvicorn main:app --reload and call curl -X POST http://127.0.0.1:8000/orders.

import time

from fastapi import BackgroundTasks, FastAPI

app = FastAPI()


def send_receipt(order_id: int, phone: str) -> None:
    time.sleep(3)  # pretend to call an SMS provider
    print(f"Receipt for order {order_id} sent to {phone}")


@app.post("/orders", status_code=201)
def create_order(background_tasks: BackgroundTasks):
    order_id = 1042  # pretend we saved the order
    background_tasks.add_task(send_receipt, order_id, "9876543210")
    return {"order_id": order_id, "status": "received"}

The response returns immediately; three seconds later the receipt line prints in the server terminal.

This is easy, but it has real limits: - The job lives inside the web server process. If the server restarts or crashes, the job is lost. - There is no retry and no record of whether it worked. - Heavy jobs steal CPU and memory from the API that serves customers.

Use it for things like writing an audit log line. For anything you cannot afford to lose, use a queue.

A job queue has three parts:

1. Producer: your API. It puts a message ("send receipt for order 1042 to this phone") on the queue. 2. Broker: the queue's storage. Here, Redis. Messages wait there safely, even if everything else restarts. 3. Worker: a separate process that takes messages and runs the function.

It is like the order slips in the kitchen. The counter (API) clips a slip on the rail (Redis) and carries on serving. The barista (worker) takes slips one by one. If the barista is busy, the slips wait. You can add more baristas without touching the counter.

Popular Python tools are Celery (the most widely used, many features), RQ and arq. In Node, BullMQ is the usual choice. We use arq here because it is small, works with async code and runs on Windows, macOS and Linux. The ideas carry over to all of them.

import asyncio
import random

from arq import Retry
from arq.connections import RedisSettings


async def send_receipt(ctx, order_id: int, phone: str) -> str:
    """Runs inside the worker process, not inside the API."""
    attempt = ctx["job_try"]  # 1 on the first try
    print(f"[try {attempt}] sending receipt for order {order_id} to {phone}")
    await asyncio.sleep(2)  # pretend to call an SMS provider
    if random.random() < 0.5:  # pretend the provider is flaky
        print("  provider timed out, will retry")
        raise Retry(defer=attempt * 5)  # wait 5s, then 10s, ...
    print(f"  receipt for order {order_id} sent")
    return "sent"


class WorkerSettings:
    functions = [send_receipt]
    redis_settings = RedisSettings()  # localhost:6379
    max_tries = 3

The job function takes ctx first (arq passes it), then your arguments. raise Retry(defer=...) asks arq to run the job again later.

from contextlib import asynccontextmanager

from arq import create_pool
from arq.connections import RedisSettings
from arq.jobs import Job
from fastapi import FastAPI, HTTPException


@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.queue = await create_pool(RedisSettings())
    yield
    await app.state.queue.aclose()


app = FastAPI(lifespan=lifespan)


@app.post("/orders", status_code=201)
async def create_order():
    order_id = 1042  # pretend we saved the order in the database
    job = await app.state.queue.enqueue_job("send_receipt", order_id, "9876543210")
    return {"order_id": order_id, "status": "received", "receipt_job": job.job_id}


@app.get("/jobs/{job_id}")
async def job_status(job_id: str):
    job = Job(job_id, app.state.queue)
    status = await job.status()
    if status.value == "not_found":
        raise HTTPException(status_code=404, detail="No such job")
    return {"job_id": job_id, "status": status.value}

create_pool connects to Redis once at startup. enqueue_job takes the function's name as a string, then its arguments, and returns immediately.

python -m arq worker.WorkerSettings

Run from the folder that contains worker.py. Leave it running.

python -m uvicorn main:app --port 8000

Use the same virtual environment.

curl -X POST http://127.0.0.1:8000/orders
curl http://127.0.0.1:8000/jobs/PASTE_JOB_ID_HERE

The first call returns at once with a receipt_job id. Use it in the second call, several times, over the next few seconds.

The order request came back instantly, while the worker did the slow, flaky part in the background. When the pretend SMS provider timed out, the job was retried later with a growing delay instead of failing the customer's order. Status deferred means "waiting for its retry time"; queued means waiting for a free worker.

Stop the worker with Ctrl+C while a job is deferred, start it again, and the job still runs: the message lives in Redis, not in the worker.

Queues usually promise at-least-once delivery: a job may run more than once (a worker crashed halfway, a retry after a timeout that actually succeeded). So write jobs to be idempotent: running twice has the same effect as running once.

Some jobs are not triggered by a request but by the clock: "email the owner a daily sales summary at 9 pm". On a single Linux server this is cron. With arq you can add cron_jobs to WorkerSettings; Celery has "beat"; cloud platforms have schedulers. Keep scheduled jobs idempotent too, because a scheduler can fire twice after a restart.

Typical questions: "How would you send emails without slowing the API?", "What happens if a worker crashes mid-job?", "What does idempotent mean and why does it matter for queues?", "Celery vs a simple thread or BackgroundTasks?" and "How do you handle a job that keeps failing?" (retry with backoff, max tries, then a dead-letter or failed list for manual review). Mention at-least-once delivery.

I am building [describe your app]. When a user does [action, e.g. places an order or uploads a video], these things happen: [list steps]. Split them into work that must finish before the HTTP response and work that should become a background job. For each background job, say what to pass in the message, how to make it idempotent, and what should happen after three failures.

What to expect: A good answer keeps the essential database write in the request, passes IDs not objects, and describes an idempotency check and a failure path.