Cost, Privacy, and Local-First AI

Two forces shape almost every real AI architecture decision: cost and privacy. A design that ignores them works in a demo and fails in production — either the bill is unsustainable or sensitive data leaves the building. A useful default that addresses both is local-first.

A routing switch inside a secure perimeter sends most AI tasks to a local server and only a thin arrow out to an expensive cloud model.

The cost reality

Cloud LLM calls are priced per token (input + output). That sounds tiny until you multiply by volume. Two patterns dominate cost:

Embeddings are generated for every chunk of every document and re-generated whenever content changes. A large knowledge base can mean millions of embedding calls.
Generation is paid per answer, and long contexts (lots of retrieved chunks) inflate the input token count on every single call.

The architect's job is to spend tokens where they buy quality and save them everywhere else.

The privacy reality

When you send text to a cloud provider, that text leaves your perimeter. For a public-sector or regulated organization, that single fact can be disqualifying for certain data. The architecture must be able to answer: exactly which data goes to which provider, and why?

Local-first as the default

"Local-first" means: if a task runs well on hardware you control, run it there; reserve cloud models for genuinely hard reasoning. This directly addresses both forces:

Embeddings run locally. They're high-volume and don't need a frontier model — a small local model (e.g. nomic-embed-text) is fast, free per call, and keeps content private. Re-indexing the whole knowledge base costs nothing but time.
Routine generation runs locally where a mid-size local model is good enough.
Cloud is reserved for advanced reasoning — the few features (deep mentoring, nuanced review) where a frontier model's quality is worth the cost and the data is non-sensitive.

The tradeoff to respect

Local-first isn't free of cost — it trades a token bill for hardware and operational responsibility (you run the model server, you keep it up). The right answer is rarely "all local" or "all cloud"; it's a deliberate routing policy that sends each task to the cheapest option that meets the quality bar, and documents the data flow so a reviewer can trust it.

LyraLearn runs embeddings and routine tutoring on local hardware, and routes only its hardest reasoning features to a cloud model — a concrete example of the policy you'll learn to design.