Embeddings and Vector Search

This is the layer the last lesson evicted from inside the LLM. It deserves its own clean mental model, because you'll use its vocabulary in every RAG conversation.

What an embedding is

An embedding model is a separate, much smaller neural network with one job: convert a piece of text into a vector — a fixed-length list of numbers (often 768 or 1,536 of them) — such that texts with similar meaning get vectors that are close together.

That's the magic property, and it's worth one beat of appreciation: "reset my password" and "I can't log into my account" share almost no words, but a good embedding model places their vectors near each other, because it learned meaning-similarity from training. Keyword search matches strings; embedding search matches meaning.

Facts that make you sound fluent:

• The vector's length (its dimension) is fixed by the model — every text, from a sentence to a page, becomes the same-size vector.
• Similarity between two vectors is computed with simple math (cosine similarity is the usual choice): a number saying how close two meanings are.
• Embeddings are cheap and fast compared to LLM calls — you can embed millions of chunks.
• The same model must embed both sides. Documents embedded with model A and queries embedded with model B land in unrelated vector spaces; nothing matches. (This also means changing your embedding model requires re-embedding the whole corpus — a real operational cost worth mentioning in design discussions.)

What a vector index/database does

A vector index stores millions of (vector → text chunk) pairs and answers one question fast: given this query vector, which stored vectors are nearest? — the top-k nearest neighbors. That's all a "vector database" fundamentally is. It can be a dedicated product, a capability inside a general database (SQL Server, Postgres), or an in-memory library — an architecture choice, not a mystery.

The honest limits (this is where interviews are won)

Naming the limits shows you've operated this, not just read about it:

Closest is not correct. Vector search always returns something — the nearest k vectors exist whether or not they're actually relevant. If the corpus doesn't contain the answer, you get confident-looking near-misses. This is over-matching, and it's the retrieval-side root of "wrong answer with a real citation." The defense is a relevance threshold: if the best similarity score is below the bar, the system should say "I don't have a grounded answer," not run with the least-bad match.

Chunking is a real design decision. Documents are split into chunks before embedding — too small and meaning fragments ("the fee is waived" loses which fee, when); too large and one vector averages several topics, matching everything weakly. Chunk size, overlap, and respecting document structure (sections, headings) measurably move retrieval quality.

Embeddings capture similarity, not truth or recency. An outdated policy chunk embeds just as beautifully as the current one. Corpus versioning — knowing which edition of every document is in the index, and re-indexing on change — is data governance, and it's on you, not the vector database.

Interview drill

1. "What's an embedding?" — A vector representing meaning, produced by a dedicated model, such that similar meanings are near each other; retrieval compares vectors, not words.
2. "What does the vector database do?" — Nearest-neighbor search over chunk vectors; returns top-k candidates for the generation step.
3. "What goes wrong?" — Closest ≠ correct (thresholds), chunking quality, stale corpus (versioning), and mismatched embedding models between index and query.