Evaluation Sets and Regression Testing
Metrics tell you that quality moved. An evaluation set tells you whether a change you're about to ship will move it — before users ever see it. This is how you stop a prompt tweak or a model upgrade from silently breaking answers.
Golden questions with expected sources
An evaluation set is a curated list of golden questions — representative, important, and sometimes deliberately challenging — each paired with what a good answer must look like. For a RAG system like LyraLearn's tutor, the expectation is usually an expected source: the lesson or document the answer should be grounded in.
A single entry might capture:
- The question ("What is p95 latency and why use it over an average?").
- The expected source (Module 13, lesson 3).
- An expectation about refusal — some questions should be refused (off-topic, no evidence), and verifying that the system correctly declines is just as important as verifying it answers.
The set lives in source control alongside the code, so it's reviewed and versioned like any test.
Re-run on every change
The discipline is simple: re-run the evaluation set whenever something that affects answers changes — a new prompt, a model upgrade, a re-indexed knowledge base, a retrieval-parameter tweak. For each golden question you check that retrieval still surfaces the expected source, that evidence strength stays strong, and that the refusal behavior matches expectation.
Because this runs in CI as a regression test, a quality drop fails the build instead of reaching production. You catch the regression in a pull request, not in a user complaint three weeks later. It is the AI equivalent of the unit tests you already trust for ordinary code.
Close the loop with the evaluation record
Evaluation sets and the ops.AiEvaluations record reinforce each other. Production data tells you
which questions are failing or refusing in the wild — and those become new golden questions,
keeping the set honest and growing with real usage. The record provides the same fields (evidence
strength, refused, latency) the eval harness asserts on, so offline tests and live monitoring speak
the same language.
Put the whole module together and you have the loop that lets a probabilistic component live safely in production: observe every call, measure the metrics that matter, and guard quality with a regression suite — all anchored to LyraLearn's own evaluation record, Serilog logs, and OpenTelemetry traces. You can't make AI perfect, but you can make it measurable — and measurable is what improvable means.