Content Safety and Moderation

A citizen-facing government chatbot that emits one hateful or dangerous reply doesn't have a bug — it has a headline. Content safety is the managed-service layer that keeps generative systems inside acceptable bounds, and it's both an AI-102 exam topic and a non-negotiable architecture component for any public-facing deployment.

The moderation categories

Azure AI Content Safety (and its equivalents at other providers) classifies text and images into four harm categories: hate, violence, sexual, and self-harm. Each category returns a severity level rather than a binary flag — Azure uses 0–7 (commonly bucketed to safe/low/medium/high).

Severity matters because the right threshold is contextual. A public-health chatbot must discuss self-harm to route someone to help, while a children's education service blocks far more aggressively. The architect's job is to set per-category thresholds deliberately, document why, and make them configuration. "Block everything above medium" is a starting point, not a policy.

Where moderation sits in the pipeline

The exam-critical point: moderation runs on both sides of the model.

• Input moderation screens what users send before it reaches your LLM — abuse, attempts to generate harmful content, material you don't want in logs.
• Output moderation screens what the model produces before the user sees it. Models are probabilistic; system prompts are guidance, not guarantees. The safety gate is the guarantee.

Alongside category classification, dedicated services detect prompt injection and jailbreaks — Azure ships this as Prompt Shields, which flags both direct jailbreak attempts in user prompts and indirect injection hidden in documents your RAG pipeline retrieves. There's also groundedness detection, which checks whether a response is actually supported by the source material — hallucination as a safety category.

In .NET terms, this is middleware: a safety check wraps the chat completion call the same way authentication wraps a controller. Fail closed — if the safety service is down, the answer is "not available right now," never "skip the check."

Blocklists, classifiers, and the human loop

Two complementary mechanisms, and the exam expects you to know when each applies:

• Blocklists — exact or pattern-matched term lists. Deterministic, instant to update, ideal for organization-specific bans (a competitor name, an unreleased product, profanity variants the classifier misses). But brittle: trivial rephrasing evades them.
• ML classification — the trained classifiers behind the category scores. They understand paraphrase and context but are probabilistic and can't be patched in an afternoon.

Production systems use both: classifiers as the broad net, blocklists as the surgical override.

The last layer is human. False positives are inevitable, and for a government service, a resident wrongly blocked from asking about benefits is a real harm. Design an appeals / human-in-the-loop path: log every block with the category and severity that triggered it, let users flag wrong decisions, and route those to reviewers whose rulings tune the thresholds. A safety layer nobody can appeal isn't safety — it's just an outage with better branding.