Indexing a Large Codebase (RAG for Repos)

"RAGify the repo" sounds like one step — run an indexer, get semantic search. In practice, indexing a large ASP.NET MVC codebase well means deciding what to index, how to chunk it, and what role the index plays next to the agent's other tools. Get those wrong and you'll own a vector database that returns confident noise.

Code chunked along structural boundaries with context headers attached, feeding a hybrid lexical-plus-semantic index that serves as an entry point for a navigating agent.

What to index — and what to skip

Index the things that carry meaning: source files, Razor views' logic, the documentation layer from the previous lesson, and — highest value per byte — short natural-language summaries of each service/controller (LLM-generated once, refreshed on change). Skip generated code, bin/obj, vendored packages, and minified assets; they add noise and cost. Remember the content-vs-container rule from Module 6: a .cshtml file is a template — index what it means, not its markup.

Chunk by structure, with context attached

Fixed-size chunks butcher code — half a method embeds as gibberish. Chunk by structural unit (class, method, view section) using Roslyn or tree-sitter, and prepend each chunk with a context header before embedding:

// File: Business/Services/DocumentStorageService.cs
// Class: DocumentStorageService | Method: UploadAsync
[code]

That header is often the only place "document storage" and UploadAsync co-occur — it's what lets a conceptual query find the right method. This is contextual retrieval, and on code it routinely matters more than which embedding model you chose.

Hybrid search, and the honest reality check

Code questions split two ways. Exact-token queries (a class name, a route, .cshtml) are lexical — BM25/grep wins. Conceptual queries ("how are claims approved?") are semantic — vectors help. Ship both and fuse the rankings; a pure-vector code index reliably disappoints. And the deeper truth from Module 9: for multi-hop questions ("how is X handled end-to-end?"), an agent navigating — grep, read, follow references — beats one-shot retrieval, because the answer spans controller → service → view. The index's real job is to give that agent a fast entry point, not to be the oracle.

Verify the pipeline before you trust it

Indexing pipelines fail in ways that perfectly imitate a weak model (Module 5's case study: a single score inversion produced three false diagnoses). Before rollout, run the cheap checks: a self-similarity test (a chunk retrieved against its own text must score as identical), a discrimination test (a known-relevant chunk must beat known-irrelevant ones), and a small eval set of real developer questions with the files you expect back. Ten questions with expected answers will tell you more than any demo — and they become your regression suite every time you re-chunk, re-embed, or change models.