The Ingestion Pipeline

Retrieval is only as good as what you put in the index. The ingestion pipeline is the out-of-band process that turns raw documents into searchable evidence: it splits each document into chunks, turns each chunk into a vector, and stores both the text and the vector so that queries can find them later. In LyraLearn this is the job that reads every lesson markdown file and prepares it for the AI Tutor — and it runs before any user asks a question, never on the request path.

Documents move along a conveyor where they are split at headings, hash-checked so unchanged chunks skip ahead, embedded, and stored with text and vector together.

Chunking: heading-aware splits

A whole document is too coarse to retrieve against — you want to return the paragraph that answers the question, not the entire file. So you split documents into chunks. Naive splitting by a fixed character count cuts sentences in half and strands ideas across boundaries. Instead, LyraLearn chunks heading-aware: each ## section becomes its own chunk, carrying its heading as context. That's why these lessons are written with frequent, descriptive subheadings — every heading is a retrieval unit. A well-titled section like "Why RAG beats fine-tuning" becomes a self-contained chunk a query can land on directly.

Good chunks are focused (one idea), self-describing (the heading travels with the text), and sized for the embedder — large enough to carry meaning, small enough to stay specific.

Embedding and storage

Each chunk is passed to an embedding model that converts text into a vector — a list of numbers capturing meaning, so that passages about the same topic land near each other in vector space. LyraLearn runs this locally with Ollama using nomic-embed-text, which produces 768-dimension vectors. Local-first embedding matters for the public sector: no document content leaves the network, there's no per-call API cost, and the pipeline runs offline.

The chunk text and its 768-dim vector are stored together in SQL Server 2025, which has a native vector column type. Keeping text and vector in the same row of the same database means retrieval is a single query — no separate vector store to operate, secure, and keep in sync.

Idempotency by content hash

Re-embedding every chunk on every run is wasteful and slow. The pipeline is made idempotent with a content hash: for each chunk, compute a hash of its text and store it alongside the vector. On the next run:

Unchanged chunk (hash matches) → skip; the existing embedding is reused.
Changed chunk (hash differs) → re-embed and overwrite.
New chunk → embed and insert.
Deleted chunk → remove its row.

So editing one paragraph of one lesson re-embeds exactly that one chunk, not the whole library. This keeps ingestion cheap enough to run often, which is what keeps the Tutor's knowledge fresh.

Out-of-band, not on the request path

Crucially, ingestion happens out-of-band — on a schedule or when content is published, as a background job. By the time a learner asks a question, the embeddings already exist and retrieval is a fast read. Never embed documents inside a user request: it's slow, it's wasteful, and it couples the answer's latency to the size of your corpus.