How LLMs Work (Enough to Architect With)

You don't need to train models to architect with them, but you do need an accurate mental model of what they are. A large language model is a function that, given a sequence of text, predicts the next token — over and over — to produce a response. Everything else follows from that.

Tokens, not words

Models read and write tokens, not characters or words. A token is roughly ¾ of a word. Three consequences matter to an architect:

• Cost and limits are measured in tokens. Pricing, context windows, and rate limits are all token-based, so estimating token counts is a real design skill.
• The context window is finite. Everything the model "sees" for one call — system prompt, retrieved context, conversation, and the answer — must fit in a fixed token budget.
• Longer context isn't always better. More tokens cost more and can dilute the signal; you want the relevant context, not all of it.

Prediction, not retrieval

The model generates each token by sampling from a probability distribution over the vocabulary. It is predicting plausible continuations, not looking up facts. This is why:

• It can produce fluent, confident text that is wrong (hallucination).
• The same prompt can yield different answers (controlled by the temperature setting).
• It has no built-in notion of "I don't know" unless you design the system to provide one.

Training vs context

A model's "knowledge" comes from two places:

• Parameters — what it learned during training, frozen at a cutoff date. Static, and not specific to your organization.
• Context — what you put in the prompt at call time. Fresh, specific, and the lever you control as an architect.

The entire discipline of RAG (Module 6) exists to exploit that second lever: rather than hoping the model memorized your facts, you retrieve them and put them in the context.

The architect's takeaway

Treat the model as a powerful but unreliable text predictor with a fixed memory and a finite attention budget. Your job is to feed it the right context, constrain its output, and check its work — not to trust it.