Evaluating AI Systems

"It looks good in the demo" is not evaluation. LLM outputs are non-deterministic and failure is often subtle — a fluent, confident answer that is quietly wrong. The fix is the same discipline we apply to any other system: define what good means, measure it repeatedly, and never let a change ship without proof.

A golden set of real questions measured at two separate stages, retrieval and answer, with an LLM judge scoring against a rubric and a human periodically calibrating the judge.

Golden sets: real questions, expected outcomes

The foundation is a golden set (or eval set): a curated collection of real questions — pulled from actual usage, not invented in a meeting — each paired with an expected outcome. The expectation doesn't have to be an exact string; it can be "must cite module 7," "must refuse," or "must mention connection pooling." In LyraLearn the raw material is already there: the ops.AiEvaluations table records every question the tutor was asked, so building the set is a SQL query plus human curation, not a research project.

Retrieval metrics vs. answer metrics

For a RAG system you must evaluate two stages separately, because they fail differently. Retrieval metrics ask: did the right chunks come back? (recall, precision, ranking). Answer metrics ask, given those chunks: is the response grounded (every claim supported by the retrieved evidence, no hallucination), relevant (actually answers the question asked), and complete (doesn't drop half the answer)? If retrieval is bad, no prompt tweak will save the answer — measuring both tells you which stage to fix.

LLM-as-judge — powerful, biased

Scoring hundreds of answers by hand doesn't scale, so a common pattern is LLM-as-judge: a strong model grades each answer against a rubric. It works — but judges carry known biases. Position bias: when comparing two answers, judges favour whichever is presented first (or last). Self-preference bias: models rate their own outputs higher than a rival's. Mitigate with explicit rubrics (score groundedness 1–5 against these criteria, not "which is better"), randomized ordering in pairwise comparisons, and using a different model as judge than the one being judged. And calibrate: periodically have humans score a sample of the same answers and check the judge agrees. Human evaluation is expensive, so spend it where it counts — as the calibration anchor, not the daily grind.

Regression evals: quality can't silently degrade

Finally, make it continuous. A regression eval runs the golden set on every meaningful change — new prompt, new model, new chunking strategy — and compares scores to the baseline. Exactly like a unit-test suite: if groundedness drops or the refusal rate spikes, the change doesn't merge. Without this, quality erodes one "harmless" tweak at a time, and nobody notices until a user does.