Retrieval and Evidence Grading

With documents chunked, embedded, and stored, the live request path has two jobs: find the passages most likely to answer the question, and judge whether they're strong enough to answer from at all. The second job is what separates a trustworthy RAG system from a confident fabricator. In LyraLearn this is the heart of what the AI Tutor does the moment you hit enter.

Hybrid search: vectors plus keywords

When a question arrives, it is embedded with the same model used at ingestion — Ollama nomic-embed-text, 768 dimensions — so the query vector lives in the same space as the chunk vectors. Retrieval then runs two complementary searches:

• Vector (semantic) search — SQL Server 2025's VECTOR_DISTANCE ranks chunks by how close their embedding is to the query's. This catches meaning even when the wording differs: "how do I stop the model making things up" finds the passage on grounding and refusal.
• Keyword boost — exact term matches (a product name, an error code, an API like VECTOR_DISTANCE itself) are boosted, because semantics alone can miss the literal token a user typed.

Combining them is hybrid search. Vectors handle paraphrase and synonyms; keywords pin down the exact terms that have to appear. The Tutor scores each candidate chunk on both and returns the top few as the evidence set.

Grading the evidence

Retrieval always returns something — the closest chunks, even when the corpus has nothing relevant. So before generating, LyraLearn grades how strongly the retrieved evidence actually supports an answer, using the distance scores and match quality:

• Strong — close, on-topic chunks directly address the question. Answer confidently.
• Moderate — relevant material, but partial or tangential. Answer, and hedge where the evidence thins out.
• Weak — only loosely related chunks came back. Answer cautiously, if at all, and say so.
• None — nothing crosses the relevance threshold. Do not answer.

Grading turns a raw similarity score into a decision the rest of the pipeline can act on. It is the gate between "we have evidence" and "we're guessing."

Fail closed: refuse when there's no grounding

The defining behaviour of a safe RAG system is what it does when grading returns None: it fails closed and refuses. The Tutor replies that the lessons don't cover the question rather than inventing an answer the model has no basis for. This is deliberate — the costly failure mode in enterprise and public-sector AI isn't "I don't know," it's a fluent, wrong answer that a user trusts.

Concretely: ask the Tutor something the lessons address and it answers with citations; ask it about today's weather and it declines. That refusal isn't a gap in coverage — it's the evidence grader working exactly as designed. Only when grading clears the bar does the request proceed to generation, where the model answers strictly from the graded evidence.