Hallucinations and Trust

"How do you stop it from making things up?" At a state credentialing agency this is not a theoretical question — a fabricated course match could tell a candidate they've met a requirement they haven't. The interviewer is testing whether you treat hallucination as a managed engineering risk with layered defenses, or hand-wave it with "the models are getting better."

Layer one: grounding and citations

The honest opening: you can't eliminate hallucination, so you design so that unsupported output can't become a decision. First defense is grounding — the model reasons only over retrieved text (the transcript, the SMR language), and every claim must carry a citation back to a specific source span. In the transcript service, a suggested match isn't "this course satisfies SMR 3.2"; it's "this course may satisfy SMR 3.2, based on this catalog description and this requirement text" — with both quoted for the analyst. If the evidence can't be shown, the suggestion doesn't ship.

Layer two: thresholds, refusal, and validation

Second defense is knowing when to say nothing. Set evidence thresholds: below a confidence floor, the system refuses to suggest and routes the item to a human as unmatched — needs review. A refusal is a correct answer; a confident guess is a defect. This is the fail-closed posture — uncertainty defaults to the safe outcome, never to "met."

Third, mechanical checks: structured-output validation. The model returns schema-constrained JSON; your C# code verifies every cited course exists in the parsed transcript, every SMR id is a real requirement, and every score is in range. A model can hallucinate a course name — your validator, which holds the actual extracted records, catches it deterministically. Cheap, boring, and it catches an entire class of fabrication.

Layer three: humans, evals, and telemetry

For consequential decisions, human-in-the-loop is the backstop, by design: AI suggests, Commission staff make the final determination. Then close the loop with measurement — a golden evaluation set of previously adjudicated transcripts, re-run on every prompt or model change, watching the false-met rate above all. In production, log override rates: if analysts start overriding a category of suggestion more often, that's your early-warning signal, visible on a dashboard before it becomes a news story.

Frame it for government explicitly: an agency must be able to explain any determination to an appellant or an auditor. "The AI said so" is not an explanation; "here is the evidence, the suggestion, and the staff member's signed determination" is.

Red flags

• "Hallucinations are mostly solved now." They are not; the design must assume them.
• Confidence displayed as decoration — a percentage in the UI with no refusal behavior behind it.
• No eval set, no override telemetry — trust asserted rather than measured.
• Letting high-confidence suggestions auto-finalize "to save time."

Practice prompts

1. Answer "how do you stop it making things up?" in two minutes, naming all three layers.
2. An analyst reports a suggested match citing a course that isn't on the transcript. Walk through which layer failed and what you change.
3. Argue for refusal-over-guessing to a stakeholder worried it will "make the AI look dumb."