Vocabulary: MLOps and Evaluation

MLOps — the operational discipline for ML/AI systems: versioning, deployment, monitoring, evaluation, and retraining, treated with the same rigor as application DevOps. Lives in: the duty statement, verbatim ("model versioning, monitoring, and retraining"). Say it: "I treat AI systems like any production system, with the AI-specific additions — versioned behavior, evals as gates, monitored drift."

Model lifecycle — the arc from selection/training through deployment, monitoring, degradation, and retirement or retraining. Lives in: MLOps. Say it: "Every model in production has a lifecycle owner — someone accountable for when it degrades."

Model registry — the versioned inventory of models: which model and version serves which feature, with what configuration, approved by whom. Lives in: MLOps governance (Azure Machine Learning provides one). Say it: "The registry answers 'what exactly is running in production' — for LLM systems that includes the prompt version."

Prompt versioning — treating prompts, retrieval settings, and model choice as versioned, diffable, reviewable artifacts — because for LLM systems, they are the behavior. Lives in: LLMOps. Say it: "Prompts are versioned and reviewed like code — a prompt change is a release, not an edit."

Evaluation (eval) — an automated test suite for model behavior: curated inputs with known-correct outputs, run before any model, prompt, or corpus change ships. Lives in: the release gate. Say it: "No eval, no deploy — the AI version of no tests, no merge."

Gold set (golden set) — the evaluation dataset built from real cases with expert-verified answers; the ruler everything is measured against. Lives in: evaluation; step one of any accuracy claim. Say it: "Before launch we build a gold set from historical determinations our experts already made — so the accuracy number exists before any user does."

LLM-as-judge — using a second model to grade the first's free-text outputs at scale, calibrated by human spot-checks. Lives in: evaluation at scale. Say it: "Exact-match scoring fails on prose, so a judge model grades outputs — and humans audit the judge."

Shadow deployment — running the new system on real inputs without showing its outputs, measuring agreement with the existing human process before anyone sees an answer. Lives in: rollout strategy; the government-friendly launch pattern. Say it: "We shadow-run beside staff decisions and build the accuracy case on production data before the public ever sees an answer."

Canary release — serving a new version to a small slice of traffic first, promoting on good metrics. Lives in: rollout strategy. Say it: "Model and prompt changes canary just like code deployments."

Drift — the world changing under a deployed model. Data drift: inputs stop resembling what you built against. Concept drift: the correct answer itself changes (policy updated). Lives in: production monitoring. Say it: "We watch both drifts — new transcript formats are data drift; a revised requirement is concept drift, and it triggers re-indexing and re-evaluation."

Retraining — refreshing a model on new data when drift erodes accuracy. For LLM-based systems, the equivalent is usually re-indexing the corpus and re-running evals on model or prompt changes. Lives in: the duty statement; the lifecycle's response to drift. Say it: "For our LLM systems, 'retraining' mostly means re-indexing and re-evaluating — same lifecycle discipline, different artifact."

Observability (for AI) — logs, metrics, and traces plus the AI-specifics: token cost per request, latency, retrieval quality, refusal rates, override rates. Lives in: production monitoring. Say it: "We track cost and quality per request the way we track latency SLOs."

Override rate — how often humans reject the AI's suggestion. Rising = degradation. Zero = usually rubber-stamping, its own alarm. Lives in: human-in-the-loop monitoring. Say it: "A suspiciously zero override rate worries me more than a moderate one — it usually means nobody's really reviewing."

Prompt caching — provider-side discount for repeated prompt prefixes; structure prompts stable-part-first (system prompt, reference material) to exploit it. Lives in: cost engineering. Say it: "We put the stable context first so the provider caches it — the variable question rides on a cached prefix."

Model routing — matching each task to the smallest model that does it well: frontier models for hard reasoning, small models for extraction and classification, local models for sensitive or high-volume work. Lives in: cost and privacy architecture. Say it: "Which model does this task actually need is the first cost question — routing is a ten-x lever."

Streaming — sending the response token-by-token as it generates, transforming perceived latency even when total time is unchanged. Lives in: latency engineering. Say it: "First words in half a second beats a five-second wait for the full block."