Small Models and Distillation
Model customization isn't only about changing a model — sometimes the architectural win is choosing a smaller one. Small language models (SLMs) in the 1–14B parameter range (Phi, Llama, Mistral, Gemma families) run on a single GPU or even a developer laptop, and for a well-scoped task they can match a frontier model at a fraction of the cost. Knowing when a 7B local model is the right architecture — not a compromise — is a differentiating exam answer.
Distillation: teacher → student
Knowledge distillation is how small models get good. A large teacher model generates high-quality outputs — answers, reasoning traces, labelled examples — and a small student model is fine-tuned on them, absorbing the teacher's behaviour on that task distribution. You are effectively using the expensive model once, at training time, to avoid paying for it on every request forever. Microsoft's Phi models were trained heavily on such synthetic "textbook-quality" data, and Azure AI Foundry exposes distillation as a managed workflow. The caveat to state in an exam: the student inherits the teacher's ability only within the distilled task — general capability still scales with size.
When a small local model is the right architecture
Three drivers, individually sufficient:
- Privacy and sovereignty. Data that must never leave your network — health records, credentials, classified material — can be processed by a model running inside it. No DPA with a model vendor, no cross-border transfer analysis.
- Cost at volume. A per-token API bill scales linearly with traffic; a self-hosted SLM is a fixed infrastructure cost. High-volume, narrow tasks (classification, extraction, embedding) cross over quickly.
- Latency and availability. No network round-trip, no vendor rate limits, works offline. Embeddings are the textbook case — LyraLearn embeds every chunk and every query with a local model via Ollama, because paying cloud latency and per-call fees for a task a 250MB model does well would be poor architecture.
Routing and cascades: the hybrid pattern
You rarely choose one model for everything. Model routing sends each request to the cheapest model that can handle it; a cascade goes further — try the small model first, and escalate to the large one when confidence is low, the query is complex, or validation fails. Done well, 80–90% of traffic never touches the expensive model.
This is precisely the platform's own hybrid local + cloud design: local models handle the always-on, high-volume work (embeddings, simple queries), while hard reasoning escalates to a cloud frontier model. In an interview, draw the spectrum from lesson one, then close with this: customization is not one technique but a portfolio — prompt what you can, retrieve what changes, tune what must be consistent, and route each request to the smallest model that can do the job.