Categories: MLOps and Evaluation

MLOps (Model Operations / Lifecycle)

Definition: the operational discipline for AI systems in production — versioning, deployment, monitoring, and retraining, run with the same rigor as application DevOps. (The duty statement's exact words: "model versioning, monitoring, and retraining.")

What's involved:

1. Model and prompt versioning — behavior artifacts tracked and reviewed like code
2. A registry — what exactly is deployed where, approved by whom
3. Release discipline — evals as gates; canary or shadow before full exposure
4. Production monitoring — cost, latency, quality signals, refusal and override rates
5. Drift watch — data drift (inputs change) and concept drift (right answers change)
6. Retraining / re-indexing — the planned response when drift erodes accuracy; for LLM systems this usually means re-indexing the corpus and re-running evals
7. Rollback and the kill switch — restore a known-good combination instantly

The opener: "Operationally this is an MLOps question — versioning, release gates, monitoring, drift, and a retraining plan. I treat AI systems like any production system, with those AI-specific additions…"

Cite it when: the question involves deploying, maintaining, updating, or "what happens after launch" for any model or AI feature.

Evaluation and Measurement

Definition: how you know an AI system works — measured accuracy before launch and monitored accuracy after, against defined metrics. (Distinguish the pair: evaluation produces the evidence; MLOps is the machinery that demands and reacts to it.)

What's involved:

1. A gold set — real historical cases with expert-verified answers
2. The worst-failure metric chosen first — which error direction must stay near zero
3. Precision and recall — wasted-work rate vs. risk-exposure rate, named correctly
4. Held-out testing — graded only on data the model never saw, most-recent period
5. Shadow running — silent operation beside the human process to measure agreement
6. Sampled human review after launch — routine, not complaint-driven
7. Override-rate monitoring — including the suspicious zero
8. Re-evaluation on every change — model, prompt, or corpus: no eval, no deploy

The opener: "That's an evaluation question, and my rule is: no gold set, no launch. Before deployment we measure against expert-decided historical cases; after, we monitor sampled reviews, override rates, and drift. Specifically…"

Cite it when: anyone asks "how do you know it works," "is it accurate enough," "prove it," or how you'd judge a vendor's model. This category IS the CIO question.

The pairing move

"Two halves: evaluation gives us the evidence — gold set before launch, sampled review and override rates after — and MLOps acts on it: versioned releases gated on evals, drift monitoring, and rollback when the numbers say so." Measurement produces, operations consumes — saying that relationship out loud is the architect framing.