Monitoring, logs and alerts

In this tutorial you will add the three pillars of observability to a small FastAPI service: structured logs, metrics and an alert. You will run the app, Prometheus (which collects metrics) and Grafana (which draws graphs) with Docker Compose, create some failures on purpose, and see them on a dashboard and in a firing alert rule.

Prerequisites: Docker with Compose (see *Docker: images and containers* and *Docker Compose for multi-service apps*) and basic Python. Everything runs on your laptop. Port numbers used: 8000 (API), 9090 (Prometheus), 3000 (Grafana).

Monitoring means watching known signals and getting told when they look wrong ("CPU above 90%"). Observability means your system gives out enough information that you can work out *why* something is wrong, even for a problem you did not predict.

Imagine a doctor. A thermometer reading is monitoring. Blood tests, scans and the patient's history are what let the doctor explain the fever. In software the three kinds of evidence are:

This lesson builds logs and metrics by hand. Traces matter once you have several services; the standard to learn is OpenTelemetry, which works with all the common back ends.

Google's SRE book recommends four signals for any user-facing service. If you only measure four things, measure these:

Averages hide pain. If 99 requests take 100 ms and one takes 10 seconds, the average looks fine but one customer is miserable. Use percentiles: p95 means 95% of requests were faster than this value.

Create a folder observe-demo with the files below. The app has a normal endpoint, a deliberately flaky one that fails about 30% of the time, and a /metrics endpoint for Prometheus.

import json
import logging
import random
import time

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse, Response
from prometheus_client import CONTENT_TYPE_LATEST, Counter, Histogram, generate_latest


class JsonFormatter(logging.Formatter):
    """Write each log line as one JSON object."""

    def format(self, record):
        data = {
            "time": self.formatTime(record),
            "level": record.levelname,
            "message": record.getMessage(),
        }
        data.update(getattr(record, "fields", {}))
        return json.dumps(data)


handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
log = logging.getLogger("brew-api")
log.addHandler(handler)
log.setLevel(logging.INFO)

REQUESTS = Counter("http_requests_total", "Total HTTP requests", ["method", "path", "status"])
LATENCY = Histogram("http_request_duration_seconds", "Request duration in seconds", ["path"])

app = FastAPI()


@app.middleware("http")
async def observe(request: Request, call_next):
    start = time.perf_counter()
    response = await call_next(request)
    duration = time.perf_counter() - start

    # Use the route template (/menu), never the raw URL, to keep label values few.
    route = request.scope.get("route")
    path = route.path if route else "unmatched"

    REQUESTS.labels(request.method, path, str(response.status_code)).inc()
    LATENCY.labels(path).observe(duration)
    log.info(
        "request handled",
        extra={"fields": {
            "method": request.method,
            "path": request.url.path,
            "status": response.status_code,
            "duration_ms": round(duration * 1000, 1),
        }},
    )
    return response


@app.get("/menu")
def menu():
    time.sleep(random.uniform(0.01, 0.15))
    return [{"id": "espresso", "name": "Espresso", "price": 120}]


@app.get("/flaky")
def flaky():
    time.sleep(random.uniform(0.01, 0.3))
    if random.random() < 0.3:
        return JSONResponse({"error": "kitchen is overloaded"}, status_code=500)
    return {"ok": True}


@app.get("/metrics")
def metrics():
    return Response(generate_latest(), media_type=CONTENT_TYPE_LATEST)

The middleware runs for every request. It records a counter (how many, by status) and a histogram (how long), and writes one JSON log line. Putting /metrics behind a normal route keeps this demo simple; in production, do not expose it publicly.

Prometheus works by pulling: every few seconds it visits /metrics on each target and stores the numbers. Create its configuration and an alert rule.

global:
  scrape_interval: 5s

rule_files:
  - alerts.yml

scrape_configs:
  - job_name: brew-api
    static_configs:
      - targets: ["api:8000"]

api is the Compose service name, which works as a host name on the Compose network. A 5 second interval is for the demo; 15 to 60 seconds is typical.

groups:
  - name: brew-api
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[1m]))
            /
          sum(rate(http_requests_total[1m])) > 0.05
        for: 1m
        labels:
          severity: page
        annotations:
          summary: "More than 5% of requests are failing"

rate(...[1m]) turns an ever-growing counter into requests per second over the last minute. for: 1m means the condition must hold for a full minute before the alert fires, which avoids paging someone for a blip.

services:
  api:
    build: .
    ports:
      - "8000:8000"

  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./alerts.yml:/etc/prometheus/alerts.yml:ro
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"

Pin image versions (for example prom/prometheus:v3.0.0 style tags, checking Docker Hub for current ones) for anything beyond a demo. Without pinning you get the latest image.

for i in $(seq 1 300); do curl -s -o /dev/null http://localhost:8000/flaky; curl -s -o /dev/null http://localhost:8000/menu; done

On Windows PowerShell use: 1..300 | ForEach-Object { curl.exe -s -o NUL http://localhost:8000/flaky; curl.exe -s -o NUL http://localhost:8000/menu }

In the Prometheus UI (http://localhost:9090/query) try these queries. This small query language is called PromQL.

Keep sending traffic for over a minute, then open http://localhost:9090/alerts. HighErrorRate should move from inactive to pending to firing, because about 30% of /flaky calls fail.

An alert in Prometheus is only a rule being true. In a real set-up Alertmanager (or Grafana alerting, or a cloud service) sends it to Slack, email or a paging tool, and someone responds.

Good alerts follow a few rules. Alert on symptoms users feel (errors, slowness) rather than every cause (a CPU spike that nobody notices). Every alert must need a human to act; if it is routinely ignored, delete or fix it, because alert fatigue makes people ignore the one that matters. Each alert should link to a runbook: a short page saying what it means and what to check first. Link this to the next lesson, *Reliability: SLOs, incidents and postmortems*, which explains how to choose alert thresholds from reliability targets.

You do not have to run Prometheus yourself. AWS has CloudWatch (logs, metrics, alarms), Azure has Azure Monitor and Application Insights, and Google Cloud has Cloud Monitoring and Cloud Logging. Managed Prometheus and Grafana services also exist. The ideas are identical: structured logs, the golden signals and symptom-based alerts. Watch the cost: log ingestion and custom metrics are billed, so do not log every debug line in production forever.

You are an on-call SRE. Below are JSON log lines from my service around the time errors began. Find the pattern: which paths and status codes are failing, when did it start, and what is the most likely cause?

Give me (1) a summary in three sentences, (2) the three next checks I should do, in order, and (3) one metric or alert I should add so I would have been told earlier. Do not guess about things the logs do not show; say what is unknown.

<logs>
[paste 30 to 100 log lines here, with secrets removed]
</logs>

What to expect: A good answer groups lines by path and status, points to a time the pattern changed and clearly separates evidence from guesses.

Typical questions: What is the difference between logs, metrics and traces? What are the golden signals? Why use p95 or p99 instead of the average? What makes a good alert? How would you find out why the site is slow? A strong answer starts from the user symptom, checks the dashboard (traffic, errors, latency, saturation), narrows by service, then reads logs or traces for the exact requests.