Pipelines and Quality

Once you know where the data lives, you need a repeatable path from source systems into the AI corpus. That path is a data pipeline, and its defining quality is not speed but trustworthiness: it must deliver the same clean result every time it runs, detect bad input before it poisons the index, and keep the corpus demonstrably fresh. Interviewers probe here because this is where AI projects quietly die — not in the model, but in a pipeline nobody can rerun safely.

ETL, ELT, and the AI corpus

Classic ETL (extract, transform, load) cleans and reshapes data before loading it; ELT lands raw data first — typically in a data lake or Azure Blob Storage — and transforms it inside the destination. For AI corpora, ELT usually wins: keep the raw documents immutable, then run transformation stages (text extraction, cleaning, chunking, embedding) as separate, re-runnable steps. When you improve your chunking strategy, you re-run one stage over preserved raw input instead of re-crawling every source system.

Validation at the gate

Quality checks belong at ingestion, not after users complain about wrong answers:

• Schema validation — does the record match the data contract? Reject and alert, don't guess.
• Deduplication — the same policy uploaded to three SharePoint sites must become one corpus document, or retrieval returns three near-identical chunks and crowds out real evidence.
• Content hashing — compute a hash of each document's content and store it. On the next run, matching hashes are skipped; only changed content is reprocessed. This is the same idempotency technique LyraLearn's ingestion uses per chunk, applied at corpus scale.

The robust pattern combines two triggers: change-driven processing (an event or webhook fires when a document is published) for low latency, plus a nightly reconciliation sweep that compares source against corpus to catch missed events and process deletions. Events keep you fresh; reconciliation keeps you correct.

Freshness SLAs and idempotency

Define a freshness SLA per source: "policy documents appear in the index within 15 minutes of publication; ticket history within 24 hours." Then measure it — record each document's source timestamp and ingestion timestamp, and alert when the gap breaches the SLA. Staleness is a silent failure mode: the pipeline looks green while the Tutor confidently cites last year's policy.

Every stage must be idempotent — running it twice produces the same corpus as running it once. Crashes and retries are normal; duplicate rows and double-embedded chunks must not be. Where a service both writes to its database and must notify the pipeline, use the outbox pattern: write the business change and an "event" row in the same transaction, and let a relay publish events from the outbox table. This closes the dual-write gap where the database commits but the notification is lost — with SQL Server, one transaction guarantees the pipeline eventually hears about every change. Exam shorthand: validate at the gate, hash for change detection, reconcile nightly, make everything idempotent.