Design: The Transcript Analyzer

This is the design round you should expect at this agency, because it's the actual job: "Design our AI-assisted transcript-review service." Run the five-step method from the last lesson, out loud, and anchor every choice in the domain.

Users, risk, data

Users and risk: EPPs submit candidate transcripts; Commission analysts review them; the output affects credentials. High-stakes, so the frame is fixed from the start: AI suggests, staff decide — human-in-the-loop is a requirement, not an option. Data: transcripts arrive as PDFs and scans of wildly varying quality; they're education records, so sensitive — processing stays inside the agency's cloud tenancy. The Subject Matter Requirements are versioned regulatory text that changes on regulation cycles, which immediately argues for retrieval over anything baked into weights.

Ingestion and matching

Ingestion: OCR via a document-intelligence service (e.g., Azure Document Intelligence) turns each transcript into structured course records — institution, course code, title, units, grade, term — each carrying a confidence score. Low-confidence extractions are flagged for human correction before analysis; garbage in at this stage poisons everything downstream. Persist the structured records in SQL Server via EF Core alongside the original document.

Matching — hybrid rules + embeddings. Don't hand the whole problem to an LLM. Deterministic rules handle what rules do best: unit minimums, required grade thresholds, articulation tables of known course-to-SMR mappings. Embedding similarity between course titles/descriptions and SMR language handles the fuzzy middle — surfacing candidate matches rules can't see. An LLM pass then assesses each candidate pair and must return evidence: the specific description text and requirement text supporting the match, plus a score. Every suggestion below the evidence threshold routes to unmatched — needs review. Fail closed.

The analyst workbench and the record

The UI is where trust is won: an MVC + Kendo workbench showing the transcript beside the SMR checklist, each suggested match expandable to its cited evidence. Analysts accept or override every suggestion, and overrides require a reason code — that's both governance and your future training/eval data. The analyst's sign-off, not the model's score, is the determination of record.

Two pieces of plumbing make this defensible to an auditor. Versioned methodology: every analysis stores the model version, prompt version, rule-set version, and SMR edition it ran against, so the agency can explain exactly how a 2026 determination was produced even after everything has been upgraded. Audit and the portable report: an append-only audit trail of suggestion → action → who → when, and a standardized, human-readable report listing met requirements and remaining gaps that travels with the candidate between EPPs.

Close with operations: a golden set of adjudicated transcripts as the regression eval, override-rate dashboards per SMR category, and per-transcript cost tracking.

Red flags

• LLM-only matching with no deterministic rule layer — unexplainable and untestable.
• No confidence gate on OCR; no evidence requirement on matches.
• Overrides without reason codes — you lose the feedback loop and the governance story.
• Forgetting the portable report; the output artifact is half the product.

Practice prompts

1. Whiteboard the full pipeline — submission to signed report — in ten minutes, method order.
2. "Why not just prompt GPT with the transcript and the SMR list?" Give the layered rebuttal.
3. An SMR edition is revised mid-year. Trace what changes through ingestion, matching, and audit.