Privacy, Lineage, and Governance

An AI corpus concentrates an organisation's most sensitive knowledge into one highly searchable place — which makes it a governance problem before it is a retrieval problem. In the public sector this is where projects are approved or killed: not "can the model answer?" but "can we prove who saw what, where the data went, and why we were allowed to use it?" Architects are expected to answer all three.

PII before the index, not after

The iron rule: personally identifiable information must be handled before content reaches an index or a cloud model — once embedded, indexed, or sent to an external API, you have lost control of it. Three techniques, in order of preference:

• Identification — scan incoming content for PII (names, national IDs, addresses, health data) using tools like Azure AI Language PII detection or Microsoft Purview classification.
• Minimization — the strongest control is not ingesting the field at all. If the Tutor never needs citizen phone numbers, exclude the column at the source.
• Redaction / pseudonymization — replace what must flow with placeholders or tokens in the pipeline, so the index and any cloud model only ever see the redacted text.

Pair this with retention policies: data deleted or expired at the source must also disappear from the corpus — chunks, embeddings, and caches included. Your nightly reconciliation sweep is also your retention-enforcement mechanism.

Lineage: citations are lineage

Data lineage answers "which source produced which chunk produced which answer." Store, with every chunk, its source system, document ID, version, and ingestion time; log, with every AI answer, the chunk IDs it retrieved. Now the user-facing citation is lineage made visible — the same metadata that lets a learner verify an answer lets an auditor trace a wrong or leaked answer back to the exact document version that caused it, and lets you invalidate every answer derived from a withdrawn source. If you cannot trace an answer to its sources, you cannot operate the system accountably.

Access control and security trimming

The index must never become a side channel around document permissions. If a user cannot open a document in SharePoint, retrieval must not surface its chunks to them — this is security trimming. Store an access-control list or security group identifier on every chunk, and filter at query time by the caller's identity (in .NET, derived from their Entra ID claims) before results ever reach the model. Trimming after generation is too late: the content has already influenced the answer.

Finally, data residency. Public-sector data is often legally required to stay within a jurisdiction or an accredited boundary. That constrains which Azure regions host your index, whether you may call a cloud model at all, and it is a core argument for local-first processing — LyraLearn's on-premises Ollama embeddings exist precisely so content never leaves the network. Exam answer in one line: govern at ingestion, trace everything, trim at query time, and know where every byte lives.