Capabilities and Limits

Designing with LLMs means knowing what they reliably do well, what they do badly, and — most importantly — where the boundary is fuzzy. Betting your architecture on a capability the model only sometimes has is a common, expensive mistake.

What LLMs are reliably good at

• Transforming text — summarizing, rewriting, translating, changing format or tone.
• Extracting structure — pulling fields out of unstructured text into JSON.
• Classifying and routing — labeling text, deciding which path a request should take.
• Drafting — first-pass code, emails, documents, that a human then refines.
• Explaining — turning a dense document or log into plain language.

Notice the pattern: they excel when the source material is in front of them and the task is to reshape or reason over it.

What LLMs are unreliable at

• Recalling specific facts — exact figures, dates, names, policies. They approximate.
• Arithmetic and counting — they predict plausible numbers, not compute them.
• Knowing their own limits — they rarely volunteer "I'm not sure" unless designed to.
• Staying current — parametric knowledge is frozen at the training cutoff.
• Consistency — the same question can get different answers.

Hallucination: the defining failure mode

A hallucination is a confident, fluent statement that is false. It is not a bug to be patched out — it is a direct consequence of how the model works (predicting plausible text). You cannot eliminate it at the model level; you manage it at the architecture level:

• Give the model the facts (grounding) instead of relying on recall.
• Require citations so claims are traceable.
• Refuse when there's no supporting evidence rather than letting the model guess.

Designing to the boundary

The practical rule: use the model for what it's good at, and put a deterministic system around what it's bad at. Need a number? Compute it in code, don't ask the model. Need a fact? Retrieve it and have the model cite it. Need a decision with consequences? Make the model's output a proposal a human or a rule approves. The capability map is the architect's design constraint.