Cost Engineering

AI features are metered by the token, and the bill scales with success — more users, more spend. Cost engineering is not "use the cheap model everywhere"; it is knowing where the money goes and spending it deliberately.

A cost-reduction funnel where caches absorb repeat requests, a router sends most remaining traffic to a cheap model and escalates little to the flagship, all under a budget meter with alert and cutoff marks.

Token economics

Providers price input tokens and output tokens separately, and output typically costs several times more. But the silent cost driver is usually input: context bloat. Every RAG chunk you stuff into the prompt, every turn of chat history you replay, every verbose system prompt is billed on every single call. A tutor that sends ten retrieved chunks when three would ground the answer pays triple for the privilege. Because LyraLearn records InputTokens and OutputTokens per call in ops.AiEvaluations, finding bloat is a GROUP BY: average input tokens per feature, sorted descending, is your optimization worklist.

Caching and batching

The cheapest token is the one you never send. Three cache layers matter:

Provider prompt caching — providers discount input tokens that repeat verbatim from recent calls. Structure prompts so the static part (system prompt, rubric, common context) comes first, and the discount applies automatically.
Semantic / response caching — if a question is (nearly) identical to one already answered, serve the stored answer. An embedding-similarity lookup in front of the model turns your most popular questions into near-zero-cost hits.
Embedding caches — never re-embed unchanged text. Key embeddings by content hash; a re-ingested document that didn't change costs nothing.

For offline work — nightly re-embedding, bulk summarization, eval runs — use batch APIs, which trade latency you don't need for a substantial discount (commonly ~50%).

Model routing and cascades

Not every call deserves the flagship model. Model routing sends each request to the cheapest model adequate for it; a cascade tries the cheap model first and escalates only when the result fails a quality check (weak evidence, low confidence, refusal). LyraLearn's split is this pattern in miniature: embeddings and light chat run on local models, and cloud spend is reserved for heavy reasoning where it actually buys quality.

Budgets, and the build-vs-API question

Give every feature a budget with quotas and alerts — per-feature spend caps that page someone at 80%, throttle at 100%. Untracked features are the ones that surprise you. Finally, build-vs-API is a TCO calculation, not a vibe: a self-hosted model has no per-token bill, but you pay in GPU hardware or rental, ops time, and evaluation burden. High-volume, stable, modest- difficulty workloads (embeddings, classification) favour local; spiky or frontier-quality workloads favour the API. Run the numbers per workload — the answer is usually both.