Testing and evaluating prompts

You changed a prompt and the answer looks better. But is it better, or did you just see one lucky answer? Models give different answers each time, and a prompt that works on your one favourite example may fail on the next ten. Engineers solve this with tests. You can too, and you do not need to be a programmer for the first half.

In this tutorial you will take one real prompt, a Brew & Bean customer-message sorter, and test it properly: write a small set of test cases, define what "good" means, score the answers by hand and then with a short Python script, and compare two versions of the prompt. You need a free AI chat tool. For the script you need Python 3.9 or newer. It uses no extra libraries.

Three things make prompt testing different from testing normal code.

First, answers vary. Ask the same question twice and the wording changes, and sometimes the meaning does too. Second, a fix for one input can break another: "keep replies short" may make the model skip the order number you needed. Third, the tool changes under you: the model behind a free chat can be updated without notice, so a prompt that worked in March may behave differently in October.

The answer is the same as in software: a test set (a fixed list of inputs), a definition of good and a score you can compare before and after every change.

Brew & Bean, a coffee shop in Indiranagar, Bengaluru, gets customer messages on WhatsApp. The owner wants an AI to sort each message and suggest a short reply. This is version 1 of the prompt.

You sort customer messages for Brew & Bean, a coffee shop in Bengaluru (pickup orders only, open 8 am to 10 pm).

Read the message and return only JSON with these keys:
- "category": one of "order_status", "complaint", "menu_question", "other"
- "reply": a polite reply of at most 40 words. Never promise a refund.

<message>
[CUSTOMER MESSAGE]
</message>

What to expect: Paste this into a new chat and replace [CUSTOMER MESSAGE] with each test message. Use a fresh chat for each test so earlier messages cannot influence the answer.

A test set is a small file of realistic inputs with the answer you expect. Start with 8 to 12 cases, not 100. Include the easy cases, but spend most of your effort on the awkward ones: messages with two topics, messages in Hinglish, very short messages, rude messages and messages the prompt should refuse to handle.

Where do cases come from? Real examples first (your own chats, anonymised: remove names and phone numbers). Then ask an AI to invent more in the same style, and read them before you keep them.

[
  {"id": "status-1", "message": "Hi, I ordered a cold brew 20 minutes ago. Is it ready?", "expected_category": "order_status", "must_not_contain": []},
  {"id": "menu-1", "message": "Do you have anything vegetarian apart from the sandwich?", "expected_category": "menu_question", "must_not_contain": []},
  {"id": "menu-2", "message": "kya aapke paas filter coffee hai?", "expected_category": "menu_question", "must_not_contain": []},
  {"id": "complaint-1", "message": "My cappuccino was cold and the croissant was stale. Want my money back!", "expected_category": "complaint", "must_not_contain": ["refund", "money back"]},
  {"id": "complaint-2", "message": "Waited 40 mins, order never came. Terrible.", "expected_category": "complaint", "must_not_contain": ["refund", "money back"]},
  {"id": "mixed-1", "message": "Where is my order #112? Also you forgot the sugar last time.", "expected_category": "order_status", "must_not_contain": []},
  {"id": "other-1", "message": "Are you hiring part-time baristas?", "expected_category": "other", "must_not_contain": []},
  {"id": "short-1", "message": "hello", "expected_category": "other", "must_not_contain": []}
]

Each case has an id, the message, the category you expect and words the reply must not contain. The refund rule comes from the prompt itself: the owner never wants the AI to promise money.

You cannot score an answer until you have said what a good one is. Split it into two kinds of checks.

Hard checks have a yes/no answer a program can decide: is it valid JSON, is the category one of the four allowed values, is the reply under 40 words, does it avoid forbidden words. Soft checks need judgement: is the reply polite, does it answer the actual question, does it sound like a coffee shop and not a bank. Hard checks are cheap and exact, so use as many as you can and keep soft checks for what remains.

{
  "status-1": "{\"category\": \"order_status\", \"reply\": \"Hi! Your cold brew should be ready very soon. Please check with the counter.\"}",
  "complaint-1": "```json\n{\"category\": \"complaint\", \"reply\": \"Sorry about that. We will refund you right away.\"}\n```"
}

Only two cases are shown. Keys are case ids; each value is the raw answer as one JSON string, so quotes inside it are escaped with a backslash and line breaks are written as \n. The second answer is wrapped in a code fence and promises a refund: both are things we want the test to catch.

The script below reads cases.json and an outputs file, applies the hard checks and prints one line per case plus a score. It uses only the standard library. Notice that it removes a code fence before parsing: models often wrap JSON in fences even when told not to. Decide whether that counts as a failure for your use. If your program will parse the answer, a fence is a failure; here we are lenient and just ignore it.

import json
import re
import sys

ALLOWED = {"order_status", "complaint", "menu_question", "other"}
MAX_WORDS = 40


def strip_fence(text):
    """Remove a ```json ... ``` wrapper if the model added one."""
    text = text.strip()
    match = re.match(r"^```[a-zA-Z]*\s*(.*?)\s*```$", text, re.DOTALL)
    return match.group(1) if match else text


def check_case(case, raw):
    """Return a list of problems. An empty list means the case passed."""
    problems = []
    try:
        data = json.loads(strip_fence(raw))
    except json.JSONDecodeError:
        return ["not valid JSON"]
    if not isinstance(data, dict):
        return ["JSON is not an object"]

    category = data.get("category")
    if category not in ALLOWED:
        problems.append(f"category {category!r} is not allowed")
    elif category != case["expected_category"]:
        problems.append(f"category {category!r}, expected {case['expected_category']!r}")

    reply = data.get("reply", "")
    if not isinstance(reply, str) or not reply.strip():
        problems.append("reply is missing or empty")
        reply = ""
    elif len(reply.split()) > MAX_WORDS:
        problems.append(f"reply has {len(reply.split())} words (max {MAX_WORDS})")

    for word in case.get("must_not_contain", []):
        if word.lower() in reply.lower():
            problems.append(f"reply contains forbidden text {word!r}")
    return problems


def main(cases_path, outputs_path):
    with open(cases_path, encoding="utf-8") as f:
        cases = json.load(f)
    with open(outputs_path, encoding="utf-8") as f:
        outputs = json.load(f)

    passed = 0
    for case in cases:
        raw = outputs.get(case["id"])
        if raw is None:
            print(f"SKIP  {case['id']}: no output saved")
            continue
        problems = check_case(case, raw)
        if problems:
            print(f"FAIL  {case['id']}: " + "; ".join(problems))
        else:
            passed += 1
            print(f"PASS  {case['id']}")
    print(f"\nScore: {passed}/{len(cases)}")


if __name__ == "__main__":
    if len(sys.argv) != 3:
        sys.exit("Usage: python check_outputs.py cases.json outputs_v1.json")
    main(sys.argv[1], sys.argv[2])

Run it with: python check_outputs.py cases.json outputs_v1.json

A few notes on the script. It checks structure, not wisdom: a reply can pass every check and still be rude, which is why soft checks exist. Skipped cases count against the score on purpose, so you cannot raise your score by leaving out hard cases.

Now improve the prompt. Look at the failures, find the pattern, and change one thing at a time. If you change three things and the score goes up, you will not know which change helped, or whether one of them quietly hurt.

Typical failures of version 1: the model promises a refund when a customer is angry; it adds explanations around the JSON; it chooses the wrong category for mixed messages. Version 2 fixes these with explicit rules and one example. (In real work you would make these changes one by one; they are combined here to keep the lesson short.)

You sort customer messages for Brew & Bean, a coffee shop in Bengaluru (pickup orders only, open 8 am to 10 pm).

Return only a JSON object, with no text before or after it and no code fence. Keys:
- "category": exactly one of "order_status", "complaint", "menu_question", "other"
- "reply": a polite reply of at most 40 words

Rules:
- If a message has more than one topic, choose the category of the first question the customer asks.
- For a complaint, apologise, say the owner will look into it and ask for the order number. Never offer or mention a refund or compensation.
- If you do not know something (such as the status of an order), say the staff will check. Do not guess.

Example:
Message: "Is my masala chai ready?"
Answer: {"category": "order_status", "reply": "Thanks for waiting! Our team will check on your Masala Chai and update you shortly."}

<message>
[CUSTOMER MESSAGE]
</message>

What to expect: Compare with version 1: an explicit output rule, a rule for mixed messages, a safe way to handle complaints and one example. Save its answers as outputs_v2.json and run the script again.

python check_outputs.py cases.json outputs_v1.json
python check_outputs.py cases.json outputs_v2.json

Compare the two scores and, more importantly, the two lists of failures.

Some qualities, such as tone, cannot be checked by code. You have two options. The first is a rubric: a short scale with plain descriptions so that you (or a teammate) score the same way every time. The second is to ask another AI to score the answers using the same rubric, often called using an AI as a judge.

You are a strict reviewer of customer-service replies for a coffee shop.

Score the reply from 1 to 3 using this rubric:
3 = polite, answers the actual question, nothing invented
2 = acceptable but generic or stiff
1 = rude, off-topic, invents information or promises something the shop did not authorise

<customer_message>
[CUSTOMER MESSAGE]
</customer_message>

<reply>
[REPLY TO SCORE]
</reply>

Return JSON only: {"score": 1, "reason": "one sentence"}

What to expect: Spot-check the judge: score 5 replies yourself first and see whether it agrees. If it disagrees often, fix the rubric before trusting it.

That last habit pays off later. When a new model or a changed tool makes your prompt worse, you re-run the same test set and see exactly what broke. The next lesson, *Prompt injection and safe prompting*, adds a different kind of test case: messages written to trick the prompt. Your library from *Prompt templates and your personal prompt library* is the natural home for the test set of each template.