Contract testing and OpenAPI validation

A contract is the agreement between an API and the people who call it: which endpoints exist, what you send, what you get back, and which status codes are possible. When the contract breaks, apps break. In this tutorial you will write the contract down as an OpenAPI document, then use tests and tools to make sure the API really keeps its promises.

You need: the Brew & Bean main.py and test setup from *Automating API tests in code* (the same folder works), Python 3.10 or newer, and about 40 minutes. Install the extra packages with pip install jsonschema openapi-spec-validator schemathesis.

The coffee shop has a mobile app and a website that both read GET /orders/{id}. A backend developer renames the field total to amount because it looks nicer. Every unit test of the backend still passes. The app now shows blank prices. Nobody tested the shape of the response that other people depend on.

Contract testing checks exactly that shape. It sits between unit tests and full end-to-end tests: cheap to run, and it catches the "I changed something small and broke someone else" bugs.

OpenAPI is a standard, machine-readable way to describe an HTTP API as a JSON or YAML document (older versions of the standard were known as Swagger). It lists every path and method, request bodies, response codes and the schema (field names and types) of each body. Tools can read it to generate documentation, client code, mock servers and tests.

FastAPI builds the document for you from your code. With the API running, open http://127.0.0.1:8000/docs for interactive docs or http://127.0.0.1:8000/openapi.json for the raw document.

curl http://127.0.0.1:8000/openapi.json -o openapi.json

Saves the contract to a file. Start the API first with uvicorn main:app --port 8000. On Windows PowerShell use curl.exe instead of curl.

Open the file and find paths (the endpoints) and components then schemas (the shapes, such as OrderOut). Those two parts are the contract.

Three tests in one file. The first checks that the document is valid OpenAPI. The second places an order and checks the real response against the OrderOut schema from the document, using the jsonschema library. The third proves the check can fail: it feeds a broken response and expects the check to complain. A check that can never fail is worthless, so always test your test.

The small schema_for helper builds a schema that points at #/components/schemas/Name and carries the document's components along, so references inside the schema can be resolved.

import pytest
from jsonschema import Draft202012Validator
from openapi_spec_validator import validate
from fastapi.testclient import TestClient

from main import app

client = TestClient(app)
SPEC = app.openapi()


def schema_for(name: str) -> dict:
    # "$ref" is resolved against the root document, so we attach "components".
    return {"$ref": f"#/components/schemas/{name}", "components": SPEC["components"]}


def assert_matches(data, name: str):
    errors = sorted(Draft202012Validator(schema_for(name)).iter_errors(data), key=str)
    assert not errors, [e.message for e in errors]


def test_spec_is_valid_openapi():
    validate(SPEC)  # raises if the document is not valid OpenAPI


def test_created_order_matches_order_schema():
    payload = {"customer_name": "Asha", "phone": "9876543210",
               "items": [{"item_id": "espresso", "qty": 1}]}
    response = client.post("/orders", json=payload)

    assert response.status_code == 201
    assert_matches(response.json(), "OrderOut")


def test_schema_check_catches_a_broken_response():
    broken = {"id": "1", "total": "free"}  # wrong type, missing fields
    with pytest.raises(AssertionError):
        assert_matches(broken, "OrderOut")


def test_documented_responses_exist():
    responses = SPEC["paths"]["/orders/{order_id}"]["get"]["responses"]
    assert "200" in responses and "422" in responses

Uses FastAPI's in-process TestClient, so you do not need the server running for this file.

Schemathesis reads your OpenAPI document and generates many requests automatically, including odd values you would not think of. It then checks that every response matches the contract: documented status code, correct body shape, no server errors. It is free and open source. Run it against the running API.

schemathesis run http://127.0.0.1:8000/openapi.json --checks all --max-examples 20

Needs the API running on port 8000. --checks all turns on every built-in check; --max-examples limits how many generated requests are tried per operation. If Windows shows a UnicodeEncodeError, set PYTHONUTF8=1 (PowerShell: $env:PYTHONUTF8 = "1") and run it again.

On the version of the API from the previous lesson, a run reported real problems. The exact wording and layout depend on your Schemathesis version, but the findings looked like this:

Read these as three different contract lies:

Undocumented 404 and 409. The code returns them, but the spec only promises 200 and 422. A mobile developer reading the spec would not handle them.

A schema-valid request was rejected. The spec said item_id is any string, so Schemathesis sent an empty one. The code then rejected it with a custom message. Either the spec is too loose or the code is too strict.

A response that violates the schema. FastAPI documents 422 errors as a list of problems, but our hand-written HTTPException(422, "...") returned a plain string instead.

The fix is to make the spec tell the truth. Declare the other status codes, and restrict item_id to the real menu ids using Literal, so the document lists the allowed values and FastAPI's own validation returns the standard 422 body. Make these changes in main.py:

ItemId = Literal[
    "espresso", "cappuccino", "cold-brew", "filter-coffee",
    "masala-chai", "croissant", "banana-bread", "veg-sandwich",
]


class OrderItem(BaseModel):
    item_id: ItemId
    qty: int = Field(ge=1, le=20)

Now the contract lists the 8 valid ids. Delete the for line in order.items loop inside create_order; validation already rejects unknown items with a standard 422.

@app.post(
    "/orders",
    status_code=201,
    response_model=OrderOut,
    responses={400: {"description": "Body is not valid JSON"}},
)
def create_order(order: OrderIn):
    ...


@app.get(
    "/orders/{order_id}",
    response_model=OrderOut,
    responses={404: {"description": "Order not found"}},
)
def get_order(order_id: str):
    ...


@app.patch(
    "/orders/{order_id}/status",
    response_model=OrderOut,
    responses={
        400: {"description": "Body is not valid JSON"},
        404: {"description": "Order not found"},
        409: {"description": "Status change not allowed"},
    },
)
def update_status(order_id: str, body: StatusIn):
    ...

The ... stands for the function bodies you already have. Only the decorator arguments change. The 400 appears because FastAPI returns it when the body is not even valid JSON, and Schemathesis sends such bodies.

Restart the API and run Schemathesis again. It should now finish without failures (you may still see a warning that some operations mostly returned 404, which only means random order ids do not exist). Your earlier pytest suite from *Automating API tests in code* should still pass too, because the behaviour for valid use did not change.

The total to amount rename would not be caught by the checks above, because the spec generated from the new code is self-consistent. To catch it you compare the new contract with the last released one. Commit openapi.json to Git as the baseline, and compare each new version against it. This small script flags removed endpoints, removed fields and new required request fields, which are the usual breaking changes:

import json
import sys


def breaking_changes(old: dict, new: dict) -> list[str]:
    problems = []
    for path, methods in old["paths"].items():
        for method in methods:
            if method not in new["paths"].get(path, {}):
                problems.append(f"removed endpoint: {method.upper()} {path}")
    for name, schema in old["components"]["schemas"].items():
        new_schema = new["components"]["schemas"].get(name)
        if new_schema is None:
            problems.append(f"removed schema: {name}")
            continue
        for field in schema.get("properties", {}):
            if field not in new_schema.get("properties", {}):
                problems.append(f"removed field: {name}.{field}")
        added_required = set(new_schema.get("required", [])) - set(schema.get("required", []))
        if name.endswith("In") and added_required:
            problems.append(f"new required request field: {name}.{sorted(added_required)}")
    return problems


if __name__ == "__main__":
    old = json.load(open(sys.argv[1]))
    new = json.load(open(sys.argv[2]))
    problems = breaking_changes(old, new)
    print("\n".join(problems) or "No breaking changes")
    sys.exit(1 if problems else 0)

Exits with code 1 when it finds a breaking change, so a CI job can fail on it.

python -c "import json; from main import app; json.dump(app.openapi(), open('openapi.new.json', 'w'), indent=2)"
python breaking.py openapi.json openapi.new.json

The first command writes the current contract from your code. The second compares it with the committed baseline. When a change is intentional and agreed, copy the new file over openapi.json and commit it.

Adding a new optional field is normally safe. Removing a field, renaming it, changing its type, making an optional request field required, or removing an endpoint is breaking. Ready-made tools exist for this comparison (search for "OpenAPI diff" tools and check which are actively maintained); the script above shows the idea and is enough for a small project.

So far the provider (the API) describes its own contract. In consumer-driven contract testing each consumer (the mobile app, the website) records what it actually uses, for example "I read id, total and status from an order". The provider's pipeline then checks it still satisfies every consumer. A provider can then change anything nobody uses, and is stopped before breaking something someone relies on. The best-known tool for this is Pact; it pays off most when many teams own many services. For a single small API, the OpenAPI checks in this lesson are the right starting point.

You are an API reviewer. Compare these two OpenAPI documents (old and new) and list every change that could break an existing client.

Group the changes into: breaking, probably safe, and needs a decision. For each breaking change, explain who is affected and suggest a backwards-compatible alternative, such as adding a new field and keeping the old one for a while.

OLD:
[paste old openapi.json or the relevant part]

NEW:
[paste new openapi.json or the relevant part]

What to expect: Check each claimed breaking change against the two documents yourself. Pay attention to removed or renamed fields, type changes, new required fields in requests, and removed status codes.

Typical questions: "What is contract testing and how is it different from integration testing?" (it checks the agreed shape of messages between two parties, without running the whole system), "How would you know an API change is breaking?", "What is OpenAPI used for?" and "What is consumer-driven contract testing?". Mention Schemathesis or Pact by name, and give the example of a renamed field that all backend tests miss.