The Analyst Workbench
The workbench is where human-in-the-loop stops being a policy statement and becomes a screen. Its job: let a Commission analyst review one submission thoroughly in minutes, with the AI's work laid out as suggestions to judge, not conclusions to rubber-stamp.
The side-by-side layout
The screen is a two-pane review surface, straight from the MVC toolkit you already have:
- Left pane: the transcript — the source document image plus the extracted, normalized coursework grid beneath it. Clicking a course highlights it in the document, so the analyst can verify extraction at a glance.
- Right pane: the SMR checklist — the domain hierarchy for the relevant subject, each node showing its status (satisfied / partial / gap / needs review) rolled up from findings.
Expanding an SMR node reveals the AI's suggested alignments: which courses, the confidence, and the evidence rationale from Lesson 3. This is a composite ViewModel served by one controller action, with jQuery wiring the cross-highlighting and a Kendo TreeView or grouped grid for the checklist. No SPA required — server-rendered MVC with targeted AJAX is exactly right here.
Accept, override, and reason codes
Every suggested finding gets an explicit analyst action:
- Accept — the suggestion becomes an analyst-confirmed finding.
- Override — the analyst rejects or modifies it and selects a reason code:
WrongDomain,InsufficientEvidence,ExtractionError,CourseLevelTooLow, plus free-text notes. Reason codes are a small lookup table, versioned with the methodology. - Add — the analyst records an alignment the AI missed (implicitly a "miss" signal).
Low-confidence suggestions arrive pre-flagged, never pre-accepted — the fail-closed rule from Lesson 1, visible in the UI. When every SMR node has a resolved status, the analyst records the determination, the audit trail captures the full accept/override history, and the submission moves to report generation.
Override telemetry: measuring the AI
Reason codes look like bureaucracy; they're actually your quality instrumentation. Because every AI suggestion ends in an explicit accept or override, you get labeled data for free:
- Override rate overall and per pipeline stage — rising overrides on embedding-sourced suggestions means the similarity threshold is too loose.
- Reason-code distribution — a spike in
ExtractionErrorpoints at Lesson 2, not Lesson 3. - Per-analyst spread — if two analysts override the same suggestions very differently, that's a methodology clarity problem for the Commission, not a model problem.
These roll up into the dashboards of Module 12; log them as structured events from day one.
Calibration and continuous improvement
Quarterly (or whenever drift shows), the loop closes: overridden findings become test cases; confirmed pairs become new Stage-1 rules; thresholds and prompts get tuned against the accumulated labels — and the changes ship as a new methodology version, leaving history intact. The system genuinely improves with use, and the humans remain, verifiably, in charge.