Monitoring and Operations

A statewide service earns trust by being boringly reliable. That takes four layers: structured logging, health checks the platform can act on, dashboards that watch the business, and runbooks so 2 a.m. problems have 2 p.m. answers.

Logging and APM

Log through ILogger<T> with a structured provider (Serilog is the de-facto standard) writing to Application Insights, and log events, not prose: SubmissionReceived, ExtractionCompleted { DurationMs, LowConfidenceFields }, FindingOverridden { ReasonCode }. Structured properties are what make the dashboards below queryable. Enrich every entry with a correlation ID that follows a submission from upload through extraction, matching, review, and report — when an EPP calls about one candidate, you filter to one ID and see the whole story.

One rule outranks the rest in this domain: no PII in logs. Log CandidateId, never names or birthdates; log "extraction completed, 41 rows," never transcript content. Logs flow to more places than databases do, and every sink inherits your weakest access control.

Health checks

Expose a health endpoint the infrastructure can probe — AddHealthChecks() in Program.cs plus MapHealthChecks("/health"), with one IHealthCheck per dependency. Check what actually fails: database connectivity, blob storage, the Document Intelligence and LLM endpoints, and the background worker's heartbeat. Report degraded vs unhealthy distinctly: AI services down is degraded (submissions flow to manual review — the fail-closed design keeps working), while database down is unhealthy (stop routing traffic). Wire the endpoint into your load balancer or App Service health probe so recovery is automatic, not paged.

The dashboards that matter

CPU graphs don't tell you the service is failing its mission. Two business metrics do:

• Queue depth per status — submissions sitting in Extracting, ReadyForReview, InReview. A growing ReadyForReview pile means you need analysts, not servers; a growing Extracting pile means the worker or a vendor API is struggling.
• Turnaround time — days from submission to determination, tracked as median and p95, because the Commission's promise to EPPs is stated in turnaround. Alert on trend, not just threshold.

Add the override-rate telemetry from Module 11 alongside them: it's the AI quality signal, and ops is where someone will actually look at it weekly.

Error triage and backup

Write runbooks while things are calm: for each alert, what it means, first diagnostic steps, and the escalation path — "poison extraction job: check dead-letter table, view vendor status page, requeue with RequeueSubmission.sql." Triage by blast radius: one submission (requeue it), one vendor (flag degraded, manual fallback), or systemic (roll back — which the one-version-compatible migrations from Lesson 2 made safe).

Finally, backups are a retention policy, not just a schedule: candidate records and audit trails keep to the state's records schedule (often seven-plus years), with periodic restore tests — an unrestored backup is a rumor. Retention, encryption at rest, and access logging on backups follow the same PII rules as the live database.