Traditional software testing assumes the same input gives the same output. Language models do not work that way, and the same prompt can produce different wording each time. Teams react in one of two ways: they stop testing and judge changes by eye, or they write brittle tests that fail whenever the phrasing changes. Neither survives contact with production.
The approach that works is evaluation: a fixed set of realistic cases, a clear definition of a good answer for each, and a repeatable way to score them.
Build an evaluation set from real usage
- Start with 50 to 200 cases drawn from real or realistic inputs, including the awkward ones: vague questions, typos, out-of-scope requests and attempts to misuse the feature.
- For each case, write down what a good response must contain, and what it must never contain, rather than one perfect answer.
- Add every production failure to the set once it is fixed, so it cannot quietly return.
Score with code first, models second
Many properties can be checked deterministically: valid JSON, a required field present, a cited source that exists, a response under a length limit, no personal data in the output. Check those with ordinary code; it is fast, cheap and exact.
For qualities code cannot judge, such as whether an answer is faithful to its sources, a second model can act as a grader, an approach often called LLM-as-a-judge. It is useful but not neutral: graders can favour longer answers or the first option they see. Give the grader a specific rubric, ask for a pass or fail with a reason rather than a score out of ten, and regularly compare its verdicts with a person’s.
Run evaluations on every change
Prompts, models, retrieval settings and tool definitions all change behaviour. Run the evaluation set in your CI pipeline whenever any of them changes, and treat a drop in pass rate like a failing test. Model providers also update and retire model versions, so pin the version you tested and re-run the set before moving to a new one.
Monitor what evaluations cannot see
- Quality signals: thumbs up or down, edits users make to drafted text, and how often a person has to take over.
- Operational signals: latency, error rates, token usage and cost per request.
- Drift: the mix of questions users ask changes over time. Sample live traffic each week and add what is new to your evaluation set.
The payoff
An evaluation set turns arguments about whether the AI got better into a number you can discuss. It also makes changing models a routine task instead of a leap of faith, which matters when better and cheaper models appear every few months.
