Design-Round Drills

Diagnostic finding: you know the components, but under pressure the answer comes out as a parts list — and the trust layer (permissions, freshness, measurement) surfaces only when a follow-up drags it out. The fix isn't more knowledge; it's a skeleton you hang every design answer on, plus drilled mechanics for the three follow-ups every document-assistant design attracts. This lesson is that skeleton and those three drills.

The skeleton (five beats, always)

1. Restate + classify. One sentence: what we're building, who uses it, and the data sensitivity. "An internal assistant over HR and policy documents — sensitive content, role-restricted, answers staff will act on." Classification drives everything after it, and saying it first is what makes the rest sound designed instead of assembled.
2. The pipeline, in phases. Corpus governance (one source of truth for documents) → ingestion (chunk, embed, store — name the stack: SQL Server 2025 native vectors, or equivalent) → query (embed, permission-filtered retrieval, grounded generation with citations, "I don't know" below threshold) → delivery (web UI behind existing auth).
3. The trust layer. Permissions enforced in code at retrieval; advisory wording with an escalation path to a human; where inference runs given the data classification.
4. Measurement. Two sentences: gold set from real staff questions before launch; sampled review, override rates, and drift monitoring after. (The full catechism, compressed — never skip it, even summarized.)
5. Alignment close. "I'd validate this design with the Systems Architect against our reference architecture before building." Ten words, scores every time.

Beats 3 and 4 are the ones that vanish under adrenaline — which is precisely why the skeleton exists. If you say the five beat-names to yourself as you answer, nothing falls out.

The quick-recall version — five words

The five beats, compressed into handles you can hold under adrenaline:

1. The What — restate it, and how hot the data is
2. The How — the pipeline, in phases
3. The Who — who sees what, and when a human steps in
4. The Matrix — measured before launch, monitored after
5. The Goodbye — "and I'd validate this with the Systems Architect"

What, How, Who, Matrix, Goodbye. The mnemonic is for your head; the panel hears the professional skins — "first, the requirement and the data's sensitivity…" — but the five words are the rails everything runs on. Two clauses that must never fall out of the compression: how hot the data is (sensitivity drives the How and the Who) and when a human steps in (the human-in-the-loop clause lives inside the Who).

Drill 1 — permission-aware retrieval (the leak question)

"How do you guarantee the assistant can't leak a document the user isn't cleared for?"

The mechanics: every chunk carries permission metadata inherited from its source document; the user's identity resolves to permission categories; and the ACL filter is applied to the retrieval query itself — unpermitted chunks never enter the candidate set, never reach the context window, and therefore cannot appear in an answer, cited or paraphrased. Citations and links get the same check.

Then name the anti-pattern, unprompted: "What you never do is enforce permissions with prompt instructions — 'don't reveal disciplinary policies' is asking the model to keep a secret it's holding. The model has no permission guards; it just generates. Enforcement lives in code, before the model sees anything." Naming the anti-pattern is worth as much as naming the pattern — it proves you know why, not just what.

Drill 2 — the freshness loop (the stale-answer question)

"HR publishes a revised policy Monday. A staffer asks Tuesday. What happens?"

The failure if you designed nothing: the index doesn't know the document changed, and the assistant serves the old policy with a confident citation. The design that prevents it has three parts:

• One source of truth for documents, so updates land in exactly one observable place.
• A re-index trigger: event-driven (publish → ingest) or a scheduled sweep that detects changes by content hash and re-chunks/re-embeds only what changed — versioned, append-only, so you can also answer "what did the policy say last month?"
• Visible freshness: answers cite the source and its as-of date — so even the worst case (a staleness window between publish and re-index) is visible to the user instead of silent.

Drill 3 — the prove-it question, compressed

"Prove it works before rollout, and tell me how we'll know if it stops working."

The two-sentence version you must be able to say even exhausted: "Before launch: an evaluation set built from real staff questions with SME-verified answers — we measure accuracy against it, and shadow-run beside the existing process. After launch: a fixed sample of answers gets routine human review, we track override and escalation rates — including a suspiciously-zero override rate, which usually means rubber-stamping — and any change to model, prompt, or corpus re-runs the evals before it ships." Expand only if asked.

The rep schedule

The skeleton and three drills are rehearsal material, not reading material. Run the full internal-assistant design out loud against the skeleton once a day until the interview — different starting emphasis each time (once leading with permissions, once with freshness, once with measurement) so the structure, not a memorized script, is what's being trained.