Handling Follow-Up Questions

Your design is on the whiteboard and the interviewer starts pushing. This phase is scored as heavily as the design itself — they're simulating the meetings you'll actually sit in. The meta-skill: stay composed, restate the constraint, place it in your framework, and adjust the design rather than defending it. Rehearse these four pushes until your answers are calm and structured.

"What if volume grows 10x?"

Don't reach for "add more servers" — decompose first. Which part is the bottleneck? Ingestion and analysis are asynchronous batch work: move them behind a queue (Azure Service Bus / Storage Queues) with a worker service, and 10x volume becomes a queue-depth and worker-count problem, not a rewrite. The real constraint is usually the model provider's rate limits — answer with provisioned throughput or quota increases, request batching, and priority tiers so interactive analyst traffic isn't starved by bulk EPP submissions. Note what doesn't need to scale: the analyst workbench is ordinary MVC + SQL Server serving modest internal traffic. Close with the honest boundary: past a threshold, the bottleneck is human reviewers, which is a staffing-and-triage conversation, not an architecture one.

"The model gets deprecated?"

This will happen — providers retire models on schedules measured in months. Two-part answer. Provider abstraction: all model calls go through one interface (IAnalysisModel); model name, endpoint, and prompt template are configuration, not scattered literals — swapping models is a config change plus a test cycle, and the methodology version stored with each analysis records which model produced what. Regression evals: the golden set of adjudicated transcripts is your safety net — run it against the successor model, compare agreement and false-met rates, and promote only when it clears the bar. Without evals, a model migration is a leap of faith; with them, it's a release.

"Answers are wrong in prod?" and "cost doubles?"

Wrong-in-prod gets an incident playbook, not improvisation: contain (thresholds up, or suggestions off entirely — the workbench degrades gracefully to manual review because HITL was the design), diagnose (audit trail + versioned methodology tell you which model/prompt/rule-set produced the bad output; reproduce it in the eval harness), remediate (fix, verify against the golden set, redeploy), and review the blast radius — which past determinations used the flawed version, and does anything need re-review. That last question is why versioning everything was non-negotiable.

Cost doubling is an engineering problem with levers, in order: measure per-transcript and per-query cost first; cache repeated work (unchanged documents don't get re-analyzed or re-embedded); route by difficulty — rules and articulation tables are free, so let a small model handle clear-cut matches and reserve the expensive model for ambiguous ones; trim prompt fat. Then present the residual as a trade-off — accuracy versus spend — for the program owner to decide with data.

Red flags

• Defending the original design instead of updating it.
• "We'd just scale horizontally" with no bottleneck analysis.
• No eval story for model migration; no playbook for production errors.
• Treating cost as beneath engineering concern.

Practice prompts

1. Have someone fire all four pushes in random order; answer each in ninety seconds.
2. Write the one-page incident playbook for "an EPP reports a wrong gap in a report."
3. Which single follow-up is weakest for you? Script and drill the ideal answer.