When Not to Use an LLM

The most common architecture mistake of the LLM era is reaching for a generative model when a classical classifier would be faster, cheaper, and more defensible. Interviewers probe this deliberately: knowing when not to use an LLM is the judgment call that separates an architect from an enthusiast.

Where classical ML wins

Prefer a trained classical model (logistic regression, gradient-boosted trees, a small neural net) when the workload has these traits:

• High volume, narrow task. Routing 500,000 documents a day into 12 categories is a classification problem. A tuned classifier scores each in milliseconds for fractions of a cent; an LLM call costs orders of magnitude more in both latency and money, multiplied by every request, forever.
• Tabular data. Claim amounts, timestamps, account histories — structured columns. Gradient-boosted trees remain the state of the art here; an LLM adds nothing but a serialization step and a bill.
• Strict latency and cost budgets. A fraud check inside a payment flow has a budget of tens of milliseconds. No hosted LLM reliably fits that envelope.
• Explainability requirements. Public-sector and regulated decisions (benefits denial, credit, hiring) often legally require you to say why. Classical models offer feature importances and stable, auditable behavior; an LLM's rationale is generated text — a plausible narrative, not an audit trail.
• Determinism and testability. A frozen classifier gives the same score for the same input, version after version. You can regression-test it like ordinary code.

Hybrid patterns: use both, each where it's strong

The real design space is rarely either/or:

1. Classifier routes, LLM handles the long tail. A cheap classifier handles the 95% of tickets it's confident about; low-confidence cases fall through to an LLM (or a human). You pay LLM prices only for the hard residue.
2. LLM as pre/post-processor. The LLM extracts structured fields from a messy email; the downstream decision is made by a classical model against those fields.
3. LLM-as-labeler bootstrapping. The classic blocker for supervised ML is "we have no labels." Use an LLM to label a few thousand historical examples (with human spot-checks), then train a cheap classifier on those labels. You get LLM-quality categorization at classifier cost — the LLM's judgment, distilled into something that runs in milliseconds.

The interview-ready heuristic

Ask three questions of any proposed LLM use: Is the output open-ended language, or a category or number? Will this run at a volume where per-call cost compounds? Must the decision be explained or reproduced later? Category/number, high volume, must-explain — that profile is classical ML. Save the LLM for what only it can do: understanding and generating language.