LLMOps — Release Discipline

A prompt edit can change system behaviour as much as a code change — yet many teams still paste prompts inline and tweak them in production. LLMOps applies ordinary release engineering to the AI-specific artifacts: prompts, models, and the configuration that binds them.

A prompt as a versioned artifact passing through an eval gate in CI, then a canary split sending a small traffic slice to the new version, with a one-flip rollback switch back to the previous version.

Prompts and models are versioned artifacts

Treat every prompt template as a versioned artifact, not a string literal buried in a service class. A prompt registry — even a simple table or config file with an id, version, template, and changelog — makes prompts config-driven: the .NET code asks for tutor_answer@v7 instead of hard-coding text. That gives you diffs, code review, and an audit trail. The same goes for models: pin exact model versions (gpt-4o-2024-11-20, not gpt-4o), because a floating alias means the provider can change your system's behaviour overnight without a single commit on your side. LyraLearn's evaluation record makes this concrete — the PromptHash and Model columns tell you exactly which prompt/model pair produced any answer, so a regression can always be traced to the artifact that caused it.

Eval suites in CI

Wire the golden set from the previous lesson into the pipeline: a prompt change runs the eval suite before merge, the same way a code change runs unit tests. The PR that "just rewords the system prompt" gets a groundedness score, a refusal rate, and a latency delta attached to it — and if the numbers regress, it doesn't merge. This is the single highest-leverage LLMOps practice: it converts prompt engineering from folklore into engineering.

Canary, shadow, rollback

Model swaps deserve the same caution as a risky deploy. In a canary deployment, route a small slice of traffic (say 5%) to the new model and compare its evaluation records — refusal rate, evidence strength, latency — against the incumbent before ramping up. In a shadow deployment, the new model receives a copy of real traffic and its answers are logged but never shown to users — zero risk, real-world data. Either way, have a rollback plan that is one config flip, not a redeploy: because prompts and models are config-driven, reverting means pointing back at the previous version.

Providers retire models — plan for it

Model deprecation is not hypothetical: providers retire models on published schedules, and an API that worked Friday can start returning errors Monday. Defence: pin versions so nothing changes silently, watch deprecation announcements, and treat each migration as a release — run the full eval suite against the successor model, canary it, then cut over. The teams that get burned are the ones who discover a retirement from a production incident instead of a calendar entry.