The Measurement Catechism

When a panel asks "how do you know your AI is accurate enough to deploy, and how will you know if it degrades?" — most candidates answer with guardrails: "it only answers from context, uncertain cases go to a human." Guardrails are good, but they answer a different question. Guardrails are how you limit harm; the question was how you know. Knowing is measurement, and this lesson drills the answer into a structure you can produce under pressure — a catechism: fixed questions, fixed answers, recited until automatic.

Before launch: three sentences

1. Build a gold set. "Before launch, we build an evaluation set from historical cases with known-correct outcomes — decisions our experts already made. The AI is measured against it: we know the accuracy number before any user sees the system." A few hundred representative cases beat a thousand easy ones — include the hard, weird, and adversarial.

2. Pick the metric that matches the worst failure. "We decide which error direction is unacceptable and measure it specifically. For an eligibility-style system, telling someone they qualify when they don't is the failure that must stay near zero — so we track false positives as the headline number and accept more 'needs human review' as the price." This is precision vs. recall wearing business clothes, and saying it this way shows you can translate.

3. Shadow-run before exposure. "We run the system silently alongside the human process — real cases, no user-visible output — and measure agreement with staff decisions. The accuracy case is built on production data before the public ever sees an answer."

After launch: three sentences

4. Sampled human review. "A fixed percentage of production outputs gets routinely reviewed by staff — not just complaints. That sample is our ongoing accuracy measurement; complaint-driven review only shows you the failures users noticed."

5. Watch the behavioral signals. "We monitor override rates — how often humans reject the AI's suggestion. Rising overrides mean degradation; a zero override rate is its own alarm, because it usually means rubber-stamping, not perfection. Alongside: refusal rates, escalation volumes, latency, and cost per request."

6. Watch for drift, and re-run the evals. "Inputs drift — new document formats, new patterns the system wasn't built against; and the ground truth drifts — policies change, so the corpus and the gold set must be versioned and re-validated. Every change to model, prompt, or corpus re-runs the full evaluation before it ships: no eval, no deploy."

Six sentences. Practice them as a unit until the structure survives interview adrenaline: gold set → worst-failure metric → shadow run → sampled review → override rates → drift and re-evaluation. Guardrails (context-only answers, human routing, kill switch) then belong in a closing sentence — as the complement to measurement, not the substitute: "and around all of it, the guardrails: advisory output, human review on low confidence, and a kill switch — but guardrails limit harm; measurement is how we know."

Answer-closing discipline

One delivery habit, because it showed up in diagnostics: never end an answer with "I'm not really sure." In a scored interview, trailing off donates your points away. If you're confident: end on the strongest sentence and stop. If you're genuinely at your edge, close with the professional version of uncertainty: "That's how I'd approach it; the part I'd validate with the team is X." That sentence converts uncertainty from a weakness into collaboration — and it's always available, because there's always an X. Practice the pivot: every rehearsal answer this week ends either on strength or on "what I'd validate is…" — never on a shrug.