Worked Design — A Citizen Services Assistant
Scenario: a state agency wants an assistant on its public website that answers citizens' questions about licenses, benefits, and filing deadlines. Run the five-step method out loud, trade-offs included. This is the classic public-sector exam scenario — and it is essentially the capstone's credential assistant with the risk turned up.
Requirements, users, risk
Users are the general public: no training, no patience, and some will probe the system deliberately. The agency's requirement is accuracy over coverage — a refusal is acceptable, a wrong deadline is not. That makes this a high-risk system: answers carry legal and financial consequences for citizens. Two requirements follow immediately: citations are mandatory (every answer links the official page it came from), and PII care (citizens will paste case numbers and SSNs into the chat box, so conversation logs are sensitive even though the knowledge base is public). Trade-off stated aloud: prioritizing accuracy means the assistant answers fewer questions than a loose chatbot would — the agency accepts that.
Data, pattern, guardrails
The ground truth is the agency's public web content — already approved by counsel, but updated continuously. That freshness profile decides the pattern: RAG, change-synced — the same change-driven re-embedding pattern as the capstone's WordPress sync, so the index follows the site within minutes instead of on a nightly rebuild. Fine-tuning loses immediately: a model fine-tuned on January's fee schedule confidently recites it in March.
Retrieval is hybrid (vector + lexical), because citizens search by exact form numbers and statute references that pure semantic similarity fumbles. On top of retrieval, evidence grading: score whether the retrieved passages actually support an answer, and refuse fail-closed when they don't, escalating to a human channel — phone number, office finder, ticket. The live credential-assistant demo from the capstone proves this exact grade-and-refuse loop in production.
Because the surface is public, guardrails run both directions: content safety on input and output (Azure AI Content Safety fits here), plus prompt-injection defenses — retrieved web content is untrusted data, never instructions, and the system prompt survives "ignore previous instructions." Trade-off: every guardrail adds latency and refusals; for a high-risk public surface, that is the correct side of the trade.
Operations
Ship a golden set — a few hundred real citizen questions with approved answers — and run it before every prompt, model, or index change. Monitor the refusal rate as a first-class metric: rising refusals mean the index has drifted from the site; falling refusals after a change may mean the grader got permissive, which is worse. Keep per-call evaluation records (question, passages, grade, answer, citations) so any answer can be reconstructed after the fact — in government, auditability is a requirement, not a nice-to-have. Cost stays modest: cheap local or small-model embeddings for sync, a capable cloud model only at answer time, cache the common questions. On a .NET stack this is one sidecar service and a queue — boring hosting, deliberate guardrails. That combination is the answer the grader is looking for.