Capacity, Latency, and SLOs

An AI feature that is right but slow, or right but down, still fails its users. This lesson closes the module with the operational contract: how fast, how available, and what happens when the model provider — a dependency you don't control — has a bad day.

An AI feature's operational contract: a streaming response that starts fast, retry-with-backoff and a circuit breaker against a flaky provider, a degradation ladder that never sacrifices truth, and an error-budget gauge governing release freezes.

Latency budgets and perceived speed

Set a latency budget per experience, not per system. An autocomplete hint has ~300 ms; a chat tutor a few seconds; a nightly quiz generator can take minutes. Budget the whole pipeline — retrieval, prompt assembly, model call, post-processing — and know which stage spends what (LyraLearn's LatencyMs column gives you the model call; traces give you the rest). For chat, the highest-leverage technique is streaming: time-to-first-token is often under a second even when the full answer takes ten, and users perceive a stream that starts immediately as fast. Ship streaming before micro-optimizing anything else.

Rate limits, retries, and thundering herds

Providers enforce rate limits (requests and tokens per minute); hit them and you get 429s. If your traffic is serious, buy provisioned throughput (reserved capacity such as Azure OpenAI PTUs) for the baseline and let pay-as-you-go absorb spikes. Handle transient failures with retry with exponential backoff and jitter — the jitter matters, because a hundred clients retrying on the same schedule after a blip create a thundering herd that re-triggers the outage they're recovering from. In .NET this is a Polly policy plus a circuit breaker that stops hammering a provider that is clearly down.

Graceful degradation, fail closed

Decide in advance what each feature does when the model is unavailable. Graceful degradation keeps the product useful: serve cached answers, fall back from the cloud model to a local one, or show retrieved sources without the generated summary. But quality fallbacks must fail closed: if retrieval returns no usable evidence, LyraLearn refuses rather than letting the model improvise — degraded service may never mean degraded truthfulness. A refusal is a controlled failure; a hallucination is an uncontrolled one.

SLOs and error budgets for AI features

Formalize all of this as SLOs. Classic ones apply — availability (99.5% of tutor requests succeed) and p95 latency (time-to-first-token under 2 s) — but AI features add quality SLOs: refusal rate stays under a threshold, and the share of answers with strong groundedness stays above one. The Refused and EvidenceStrength columns make both directly measurable with SQL. Each SLO implies an error budget — the tolerable miss allowance per window. Budget intact: ship model swaps and prompt changes freely. Budget burned: freeze risky changes and fix reliability first. That single rule turns "is the AI okay?" from a feeling into a policy.