The AI Threat Model

Before you can secure an AI feature, you have to name what you're protecting and what can go wrong. A traditional threat model asks "who can reach this endpoint and what can they do?" An AI threat model adds a harder question: the system now processes untrusted natural language as if it were intent, and that text can come from a user, a retrieved document, or a tool result. An AI architect builds the threat model first — everything else in this module is a response to it.

Threat-model diagram with three protected assets at the center and arrows of distinct attack types pressing in from outside.

Assets to protect

Three assets matter in almost every enterprise deployment:

Data. The records you send to a model (PII, case files, internal documents) and the knowledge base behind retrieval. This is the thing regulators and citizens care about most.
Model behavior. The system's instructions and guardrails — the "personality" and policy that keep it on-task. If an attacker can rewrite behavior, they own the feature.
The audit trail. Who asked what, what the model saw, what it answered. In public-sector work this is not optional: you must be able to reconstruct any decision after the fact.

If you can't say which of these a given control protects, it probably isn't pulling its weight.

Threats to anticipate

Prompt injection. Hostile text — direct from a user or hidden in retrieved content — that hijacks the model's instructions. The defining LLM risk; Module 11's next lesson goes deep.
Data exfiltration. Coaxing the model to reveal data it was given access to, or to smuggle secrets out through its output.
Model and output abuse. Using the feature to generate harmful, biased, or off-policy content, or to launder a prohibited action through a plausible answer.
Knowledge-base or training-data poisoning. Planting malicious content in a source the system ingests, so the attack fires later when that content is retrieved. A supply-chain attack on your data, not your code.
Denial of wallet. Driving cost or latency through expensive prompts, huge context, or request floods until the budget — or the service — breaks.

Mapping to expectations

Public-sector and enterprise reviews increasingly expect AI features to map to recognized controls — NIST-style confidentiality, integrity, availability, and the NIST AI RMF's govern / map / measure / manage loop. The value of a written threat model is that it lets you point at each control and say which asset it protects and which threat it answers — exactly what an assessor will ask.

How LyraLearn applies this

LyraLearn names its assets explicitly: lesson data and embeddings (data), the Tutor's system instructions (behavior), and the request/answer log (audit trail). Its local-first embeddings keep most data on-platform, its allow-listed ingestion resists poisoning, and per-request limits cap denial-of-wallet exposure. The platform is its own worked example — every later lesson traces back to a named threat here.