The AI Evaluation Record
Logs scroll away and traces expire. To actually study your AI over weeks and months, you need a
durable, queryable record — one row per AI call. In LyraLearn that record is the
ops.AiEvaluations table, and it is the backbone of everything in this module.
One row per call
Every time a feature invokes the model, it writes a single row capturing what happened. The columns are chosen so that later you can slice quality, cost, and performance without going back to raw logs:
- Feature — which capability made the call (
rag_tutor,quiz_generator,summarizer). - Provider and Model — e.g. local Ollama vs. Azure OpenAI, and the exact model id.
- PromptHash — a hash of the rendered prompt, so you can group identical prompts and detect when a template changed.
- InputTokens / OutputTokens — the raw material for cost and latency analysis.
- LatencyMs — wall-clock time for the call.
- EvidenceStrength — how well-grounded the answer was in retrieved sources (none / weak / strong).
- Refused — a boolean flag set when the system declined to answer (low confidence, no evidence, off-topic).
- TraceId — the correlating id that ties this row back to the OpenTelemetry trace and Serilog events for the same request.
Why each field earns its place
Nothing here is decorative. EvidenceStrength and Refused are the quality signals — together they tell you whether the RAG pipeline is finding good sources or quietly degrading. PromptHash lets you A/B a prompt change: filter to the old hash and the new one, compare refusal and latency. Tokens drive the spend dashboards. Provider/Model let you compare a cheap local model against a cloud one on the same questions. And TraceId is the thread that stitches the durable record back to the live telemetry — when one row looks wrong, you can pull the full trace and read exactly what was retrieved and sent.
Write it at the call site, fail closed
The record is populated in the same code path that makes the call, inside the same unit of work, so it can never drift out of sync with reality. Crucially, a refusal still writes a row — refusals are data, not silence. If the model call itself throws, the surrounding handler still records the attempt with the error captured.
This is the practical embodiment of "measure everything": a real .NET table you can query with plain SQL, feeding the metrics and regression tests in the lessons that follow.