Test the API with AI-written tests
Write an automated test suite for the order API with pytest and FastAPI's TestClient. AI drafts the tests, you review them with a checklist, run them and prove they catch real bugs.
You have an API that works when you click around in /docs. But "it worked when I tried it" does not scale: every change might break something you forgot to try again. Automated tests are small programs that call your API and check the answers, in under a second, as often as you like.
In this lesson you will get AI to write the tests, review them like a skeptic (AI tests can look impressive and check nothing), and run them. You end with about 27 passing tests, and you will prove that they really catch bugs.
You need: the API with the database from the last lesson, and pytest and httpx installed (they are in requirements-dev.txt).
Why tests matter, especially with AI
When AI writes or changes code, tests are your safety net. You do not need to re-read every line after each change: you run the tests. They also work as a living copy of your contract: API.md says what must happen, and the tests check that it does. Professional teams run tests automatically on every change, so a broken change never reaches the live site.
What you will do
- 1Understand the tools: pytest, TestClient and fixtures.
- 2Decide what to test.
- 3Prompt the AI for tests.
- 4Review the tests, then run them.
- 5Break the code on purpose and check that tests fail.
- 6Write one test yourself.
Step 1: The tools
Words you will meet
| Tool | What it does |
|---|---|
| pytest | Finds functions named test_something in files named test_*.py, runs them and reports pass or fail |
| assert | A line that must be true. If it is false, the test fails |
| TestClient | Part of FastAPI. Sends requests to your app inside the same program, with no server and no network |
| fixture | A reusable piece of setup, such as a clean database. Tests ask for it by naming it as a parameter |
| parametrize | Runs one test function many times with different inputs |
| monkeypatch | Temporarily changes something, such as an environment variable, and restores it afterwards |
The most important idea: tests must never touch your real database. If a test wrote to brew.db, it would fill your data with test orders, and one test could affect another. So each test gets its own empty in-memory database. FastAPI makes this easy with app.dependency_overrides: you replace get_db with a version that uses the test database.
Step 2: What to test
Use your contract as the list. For every endpoint test the happy path, then each rule and each error code. A table keeps you honest:
Test plan
| Area | Tests |
|---|---|
| GET /menu | Returns 8 items with the right prices |
| POST /orders | Total is computed by the server (2 x 120 + 150 = 390) |
| POST /orders | Client cannot set price, total or status |
| POST /orders | pickup_time is in the future and has a timezone |
| POST /orders | Bad phone, bad name, empty items, qty 0 or 21, duplicate item, bad pickup: all 422 |
| POST /orders | Unknown item is 422 |
| GET /orders/{id} | Returns exactly what was saved. Unknown id is 404 |
| PATCH status | No key or wrong key is 401. Server without a key is 503 |
| PATCH status | The full flow works. Skipping steps, late cancels and changing a final state are 409 |
| PATCH status | Unknown status is 422. Unknown order is 404 |
Step 3: Prompt the AI
Write pytest tests for my FastAPI order API.
Setup rules:
- Use fastapi.testclient.TestClient.
- Create tests/conftest.py with a fixture `client` that builds a fresh in-memory SQLite database for every test (sqlalchemy StaticPool, check_same_thread=False), creates the tables with Base.metadata.create_all, and overrides the get_db dependency with app.dependency_overrides. The fixture must set the ADMIN_KEY environment variable with monkeypatch. Never touch brew.db.
- Add fixtures `admin` (headers dict) and `valid_order` (a valid request body).
- Put the tests in tests/test_api.py. Use pytest.mark.parametrize where many inputs share one expectation.
- Test every rule and status code in the contract, not only the happy path. Every test needs at least one specific assert on a value (not just a status code where a value matters).
- Do not change the app code. If you think the app has a bug, tell me instead.
Contract:
[paste API.md]
App code:
[paste app/main.py, app/schemas.py and app/db.py]What to expect: You should get a conftest.py and a test file. Do not accept tests that only check status_code == 200.
Step 4: Review the tests
AI-written tests have typical weaknesses. They may only check the status code, copy the expected value from the code (so the test can never fail), or test the happy path only. A test that cannot fail is worse than no test, because it gives false confidence. Review with this list.
Test review checklist
- Each test has a clear name that says what must be true.
- The test checks values (total == 390), not just that something came back.
- Expected values are written by hand from the contract. They are not calculated with the same code as the app.
- Error cases are tested too: bad input, unknown ids, wrong key, wrong status move.
- Tests do not depend on each other or on order. Each starts from an empty database.
- Tests never use the real database file.
- No test is marked skip or xfail to make the run green.
Create an empty file tests/__init__.py so that Python and pytest treat the folder as a package and can import app from the project root. Then add the two reference files. First the shared setup:
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.pool import StaticPool
from app.db import Base, get_db
from app.main import app
ADMIN_KEY = "test-admin-key"
@pytest.fixture()
def client(monkeypatch):
"""A client that talks to a fresh, empty database for every test."""
monkeypatch.setenv("ADMIN_KEY", ADMIN_KEY)
# One in-memory SQLite database, shared by every connection in this test.
engine = create_engine(
"sqlite://",
connect_args={"check_same_thread": False},
poolclass=StaticPool,
)
Base.metadata.create_all(engine)
TestingSession = sessionmaker(bind=engine, autoflush=False)
def override_get_db():
db = TestingSession()
try:
yield db
finally:
db.close()
app.dependency_overrides[get_db] = override_get_db
yield TestClient(app)
app.dependency_overrides.clear()
engine.dispose()
@pytest.fixture()
def admin():
return {"X-Admin-Key": ADMIN_KEY}
@pytest.fixture()
def valid_order():
return {
"customer_name": "Asha",
"phone": "9876543210",
"items": [
{"item_id": "espresso", "qty": 2},
{"item_id": "croissant", "qty": 1},
],
"pickup": "in_15",
}The client fixture gives every test a new empty database and a TestClient. yield means: run the test now, then clean up afterwards.
Notice StaticPool. An in-memory SQLite database normally disappears when its connection closes, and a pool may open several. StaticPool forces one shared connection, so the tables you create are the ones the app uses. This is a classic gotcha that AI sometimes gets wrong.
from datetime import datetime, timezone
import pytest
def place(client, order):
response = client.post("/orders", json=order)
assert response.status_code == 201
return response.json()
# ---- GET /menu -------------------------------------------------------------
def test_menu_returns_all_eight_items(client):
response = client.get("/menu")
assert response.status_code == 200
menu = response.json()
assert len(menu) == 8
prices = {item["id"]: item["price"] for item in menu}
assert prices["espresso"] == 120
assert prices["filter-coffee"] == 80
# ---- POST /orders ----------------------------------------------------------
def test_create_order_computes_total_on_the_server(client, valid_order):
order = place(client, valid_order)
# 2 x Espresso (120) + 1 x Butter Croissant (150)
assert order["total"] == 390
assert order["status"] == "received"
assert order["customer_name"] == "Asha"
assert order["items"][0] == {
"item_id": "espresso", "name": "Espresso", "price": 120, "qty": 2,
}
def test_client_cannot_set_the_price_or_status(client, valid_order):
valid_order["total"] = 1
valid_order["status"] = "picked_up"
valid_order["items"][0]["price"] = 1
order = place(client, valid_order)
assert order["total"] == 390
assert order["status"] == "received"
def test_pickup_time_is_in_the_future_and_has_a_timezone(client, valid_order):
order = place(client, valid_order)
pickup = datetime.fromisoformat(order["pickup_time"])
assert pickup.tzinfo is not None
assert pickup > datetime.now(timezone.utc)
def test_name_is_trimmed(client, valid_order):
valid_order["customer_name"] = " Asha "
assert place(client, valid_order)["customer_name"] == "Asha"
@pytest.mark.parametrize(
"change",
[
{"phone": "12345"},
{"phone": "5876543210"}, # must start with 6, 7, 8 or 9
{"phone": "98765 43210"},
{"customer_name": "A"},
{"customer_name": " "},
{"items": []},
{"items": [{"item_id": "espresso", "qty": 0}]},
{"items": [{"item_id": "espresso", "qty": 21}]},
{"items": [{"item_id": "espresso", "qty": 1},
{"item_id": "espresso", "qty": 1}]},
{"pickup": "tomorrow"},
],
)
def test_bad_orders_are_rejected(client, valid_order, change):
valid_order.update(change)
assert client.post("/orders", json=valid_order).status_code == 422
def test_unknown_menu_item_is_rejected(client, valid_order):
valid_order["items"] = [{"item_id": "pizza", "qty": 1}]
response = client.post("/orders", json=valid_order)
assert response.status_code == 422
assert "pizza" in response.json()["detail"]
# ---- GET /orders/{id} ------------------------------------------------------
def test_get_order_returns_what_was_saved(client, valid_order):
created = place(client, valid_order)
response = client.get(f"/orders/{created['id']}")
assert response.status_code == 200
assert response.json() == created
def test_get_unknown_order_is_404(client):
assert client.get("/orders/does-not-exist").status_code == 404
# ---- PATCH /orders/{id}/status ---------------------------------------------
def test_status_changes_need_the_admin_key(client, valid_order):
order = place(client, valid_order)
url = f"/orders/{order['id']}/status"
body = {"status": "preparing"}
assert client.patch(url, json=body).status_code == 401
assert client.patch(url, json=body, headers={"X-Admin-Key": "wrong"}).status_code == 401
def test_status_moves_through_the_whole_flow(client, valid_order, admin):
order = place(client, valid_order)
url = f"/orders/{order['id']}/status"
for status in ["preparing", "ready", "picked_up"]:
response = client.patch(url, json={"status": status}, headers=admin)
assert response.status_code == 200
assert response.json()["status"] == status
assert client.get(f"/orders/{order['id']}").json()["status"] == "picked_up"
@pytest.mark.parametrize(
"path, bad_next",
[
([], "ready"), # cannot skip "preparing"
([], "picked_up"),
(["preparing", "ready"], "cancelled"), # too late to cancel
(["cancelled"], "preparing"), # cancelled is final
],
)
def test_invalid_status_moves_are_409(client, valid_order, admin, path, bad_next):
order = place(client, valid_order)
url = f"/orders/{order['id']}/status"
for status in path:
assert client.patch(url, json={"status": status}, headers=admin).status_code == 200
response = client.patch(url, json={"status": bad_next}, headers=admin)
assert response.status_code == 409
def test_unknown_status_value_is_422(client, valid_order, admin):
order = place(client, valid_order)
response = client.patch(
f"/orders/{order['id']}/status", json={"status": "teleported"}, headers=admin
)
assert response.status_code == 422
def test_status_update_for_unknown_order_is_404(client, admin):
response = client.patch(
"/orders/nope/status", json={"status": "preparing"}, headers=admin
)
assert response.status_code == 404
def test_server_without_admin_key_refuses_changes(client, valid_order, monkeypatch):
order = place(client, valid_order)
monkeypatch.delenv("ADMIN_KEY")
response = client.patch(
f"/orders/{order['id']}/status",
json={"status": "preparing"},
headers={"X-Admin-Key": "anything"},
)
assert response.status_code == 50327 test runs from 15 test functions. The parametrized tests expand into one run per case.
Step 5: Run the tests
pytestRun from the project folder with the virtual environment active. Add -v to see each test name, -x to stop at the first failure, or -k status to run only tests with status in the name.
============================= test session starts =============================
collected 27 items
tests/test_api.py ........................... [100%]
============================== 27 passed in 0.74s ==============================You may also see a warning that httpx is used with the Starlette test client in a deprecated way and suggests a newer package. This depends on the versions you install. It is only a warning: the tests still run. If you want, read the message and ask your AI how to follow it.
Step 6: Prove the tests can fail
This is the step most beginners skip. A test suite is only useful if it fails when the code is wrong. Do this experiment: change the app on purpose, run pytest, and watch the right tests fail. This is a simple form of mutation testing. Always change the code back afterwards.
Break it, then fix it
| Break this in app/main.py | Which tests should fail |
|---|---|
| Change total += item.price * line.qty to total += item.price | test_create_order_computes_total_on_the_server |
| Change "picked_up": set() in NEXT_STATUS to "picked_up": {"received"} | Nothing in our file checks this directly. A gap in the tests (see the exercise) |
| Remove the line require_admin(x_admin_key) | test_status_changes_need_the_admin_key |
| Change allow_origins to ["*"] | Nothing fails. We do not test CORS yet (see the stretch exercise) |
| In schemas.py change le=20 to le=50 on qty | test_bad_orders_are_rejected (the qty 21 case) |
The second and fourth rows show something important: passing tests do not mean the code is correct, only that the things you tested are right. Finding what is untested is the next step in every real project.
Never let AI "fix" a test just to turn it green
If a test fails, read the failure message first. Do not paste the failing test and tell the AI to "make it pass". The AI may change the test to match a bug. Decide which is right: the contract, the code or the test. Then change that one.
A pytest test fails. Explain in simple words why it fails, and tell me whether the bug is more likely in the test or in the app code. Do not change anything yet.
Contract rule:
[paste the rule from API.md]
Test:
[paste the test]
App code involved:
[paste the function]
Failure output:
[paste the pytest output]What to expect: A good answer compares the test and the code with the contract rule, and says which one disagrees with it.
Common problems
Troubleshooting
| Problem | Cause and fix |
|---|---|
| ModuleNotFoundError: No module named 'app' | Run pytest from the project folder, and make sure tests/__init__.py and app/__init__.py exist. |
| no such table: orders in tests | The test database was created on a different connection. Use poolclass=StaticPool as in conftest.py. |
| Tests pass only when run in a certain order | A test depends on another's data. Make each test create what it needs. |
| Status tests return 503 | ADMIN_KEY is not set. The client fixture sets it with monkeypatch. |
| A brew.db file appears after running tests | The override is missing, so the real database was used. Check the dependency_overrides line. |
How this comes up in interviews
Interviewers ask "How do you test an API?" A strong answer covers: unit tests for logic, API tests for endpoints with a separate test database, testing error cases and not only the happy path, and running tests automatically in CI. Another common question is "How do you know your tests are good?". The answer is the experiment you just did: break the code and see whether a test fails.
Try it: Fill the gap: final states
The experiment showed that no test checks that picked_up and cancelled are final. Write test_final_states_cannot_change that moves one order to picked_up and tries every other status, expecting 409 each time. Then break NEXT_STATUS again and confirm your test fails.
Try it: Stretch: test CORS and run tests automatically
1) Write a test that sends a request with the header Origin: http://localhost:5173 to GET /menu and checks the response header access-control-allow-origin, and a test that an unknown origin does not get it. 2) Ask your AI for a GitHub Actions workflow that runs pytest on every push, then read it with the review mindset before you commit it.
Key takeaways
- Automated tests are your safety net when AI changes code. They also check your contract.
- Tests must never use your real database. Give each test a fresh in-memory one and override get_db.
- Test errors and rules, not only the happy path. Check specific values.
- Review AI tests for asserts that cannot fail. Prove tests work by breaking the code on purpose.
- A green run only means the tested behaviour is right. Look for what is not tested.
- When a test fails, decide whether the contract, the code or the test is wrong. Never just make it green.
Check yourself
Finishing the quiz marks the lesson as done.
1. Why does conftest.py give every test its own in-memory database and override get_db?
2. An AI-written test only asserts response.status_code == 201 for a new order. What is the main weakness?
3. You break the code on purpose, run pytest, and every test still passes. What does that tell you?
4. A test fails after an AI change. What is the right first move?
5. What does @pytest.mark.parametrize do?
Saved on this device. Create a free account to keep it on every device.
Practise this with Sam
A real mock interview, out loud, with an honest scorecard. Free to start.