Categories: Operations, Cost — and Using the Frames

Operations and Observability

Definition: keeping systems visibly healthy in production — monitoring, alerting, diagnosis, and recovery — with AI adding its own signals on top of the classics.

What's involved:

1. Health monitoring and heartbeats — liveness proven, not assumed
2. Structured logging and audit trails — reconstruction is always possible
3. Alert quality over alert volume — the alert-fatigue lesson: unread alerts equal no alerts
4. Silent-failure detection — baselines catch runs that "succeed" while doing nothing
5. MTTD/MTTR as the yardsticks — detection and recovery time, measured
6. Runbooks — diagnosis knowledge in documents, not just heads
7. AI-specific signals — cost per request, latency, refusal and override rates
8. Graceful degradation — features toggle off; the platform survives its dependencies

The opener: "Operationally, my bar is that failure announces itself — monitoring, structured audit trails, baselines that catch silent failures, and measured time-to-detect. I built our operations platform around exactly this, so concretely…"

Cite it when: anything about production incidents, monitoring, reliability, "how would you support it," or what happens when things break.

Cost and Capacity

Definition: engineering AI features so the economics work at scale — because per-token pricing means design decisions are spending decisions.

What's involved:

1. Token economics — pay per token, output costs more than input
2. Model routing — the smallest model that does the task well; a 10x lever
3. Caching — prompt-prefix caching, embed-once-per-version, answer caching
4. Batching — non-interactive work goes to cheap models off the clock
5. Streaming — perceived latency is a design choice
6. Local vs. cloud trade — near-zero marginal cost vs. hardware and ops
7. Cost observability — spend per request tracked like latency

The opener: "There's a cost-engineering dimension here: the first question is which model this task actually needs, then caching and batching. At volume, routing is a ten-x lever, so…"

Cite it when: scale numbers appear ("10,000 queries a day"), budgets are mentioned, or a design choice differs at volume.

Routing mechanics — if they probe

The definition alone won't survive a follow-up like "how would routing actually work?" The key insight to lead with: the router is your code, not the AI. The model is just a string parameter on the API request — routing means your backend decides which string to send before the call goes out. Three patterns, in increasing sophistication:

1. Rule-based (static) — an if/switch in the service layer keyed on request type. When the endpoint or button already tells you the shape of the request, no intelligence is needed to route it. Deterministic code; you can unit test it.
2. Classifier-based — a small, cheap model reads the message first and returns one label ("simple FAQ," "status lookup," "transcript analysis"); your code maps each label to a model. The classification call costs almost nothing because the output is one word.
3. Cascade (try cheap, escalate) — send to the cheap model first with permission to say "I'm not confident." If the response fails a check — a can't-answer signal, a validation rule, a confidence threshold — your code retries the same request on the stronger model. You pay double only on the hard minority.

Mapped onto the transcript chatbot: an FAQ like "what can I do with a multiple-subject credential?" goes to the cheap tier (or cached answers); "what's my application status?" isn't an LLM problem at all — a database lookup, with the model at most phrasing the result; "here's my transcript — what do I qualify for?" is the expensive tier, deliberately — structured extraction plus requirements retrieval plus reasoning across pathways, where accuracy failures have real consequences.

Two guardrail clauses that turn a good routing answer into a full-credit one:

• Routing happens inside the sanctioned menu. The router picks between models approved for the data's sensitivity — cost routing never overrides the data-classification gate. (This welds routing to the What and the Who.)
• Each tier gets its own eval set. The danger of routing is silently downgrading quality to save money — a small gold set per tier catches a cheap tier slipping before users notice. (The Matrix, showing up inside a cost question.)

The dictation-pace version: "Model routing means my code selects the model per request — the model is just a parameter. Simple, high-volume queries go to a small, cheap model; the analysis path, where accuracy matters, goes to the capable model. The decision can be a rule, a cheap classifier call, or a try-cheap-then-escalate cascade. Two guardrails: routing only selects among models approved for the data's sensitivity, and each tier is measured against its own evaluation set so cost savings never silently degrade quality."

Using the frames — the technique itself

The move, every time: (1) Name the category in your first sentence — it tells the panel which rubric section you're answering and starts the note-taker's heading. (2) Define it in one line — proves the label isn't a buzzword. (3) Enumerate what's involved — numbered, at dictation pace; each item is a potential rubric match. (4) Then go concrete with your story. (5) Close by restating the category — the last thing written down is your keyword.

Most real questions touch two or three categories — say so explicitly: "There are three layers to this: a responsible-AI layer, a data-architecture layer, and an operations layer — let me take them in order." That sentence alone structures the entire answer, gives the note-taker three headings, and buys you thinking time while sounding maximally organized. That's the whole trick: the categories are your table of contents, spoken aloud.

Drill: take any past quiz question from batches 1–3, and before answering, say only the category sentence — which frames apply, in what order. Ten reps of just openers builds the reflex faster than ten full answers.