Retrieval, embeddings and retrieval-augmented generation

Why it matters

Most LLM applications over private knowledge are retrieval systems with a generator on top, and many of their failures are retrieval failures: the passage that answers the question never reached the top k. A RAG design question tests more than architecture: the revealing part is explaining how you would evaluate and debug the system.

Core ideas

Lexical retrieval. An inverted index maps terms to the documents containing them. BM25 scores a document by summing, over query terms, an inverse-document-frequency weight times a saturating function of term frequency, with a correction for document length relative to the average; the parameters k1 and b control saturation and length normalization. BM25 is strong on exact identifiers, names, error codes, rare terms and unfamiliar domains, needs no training, and is a demanding baseline.

Dense retrieval. A bi-encoder embeds queries and passages independently, and relevance is cosine or dot-product similarity. Passage vectors are precomputed, so search is a nearest-neighbor lookup, usually approximate: graph-based or quantized indexes give up a little recall for large speedups. Dense retrieval captures paraphrase and topical similarity, but can miss exact strings, handles negation and numbers poorly, and degrades outside its training domain. Retrievers are trained contrastively, pulling queries towards relevant passages and away from negatives; hard negatives, passages that look relevant but are not, matter a great deal.

Hybrid retrieval and fusion. Run both retrievers and merge the lists. Raw scores live on different scales, so normalize them or use reciprocal rank fusion, score(d) = sum over retrievers of 1 / (k + rank(d)), with k commonly set to 60. Hybrid is a sensible default when queries mix keywords and natural language.

Reranking. A cross-encoder reads query and passage together and outputs a relevance score. Because tokens attend across query and passage it is much more accurate than a bi-encoder, but it needs one forward pass per candidate, so it only reranks the top tens of candidates from the first stage.

Chunking. Chunks should be small enough to be specific and to fit several into the context, and large enough to carry meaning. Split along document structure (sections, paragraphs, functions) rather than fixed character counts where possible, use modest overlap, and attach metadata: title, section path, source, date and permissions. Chunks that depend on surrounding text ("the policy above") are hard to retrieve; prepending document or section context before indexing can help.

Grounding and provenance. Pass passages with identifiers and require citations to them; check that cited passages exist and actually support each claim (string overlap or an entailment check). Let the system say it cannot answer when the context is insufficient, and evaluate that behavior. Long context windows do not remove the need to retrieve well: studies have found that models use information in the middle of long inputs less reliably than at the beginning or end, and cost grows with every token.

Permissions. Enforce access control in the retrieval query itself (tenant and document ACL filters), not by instructing the model to ignore documents. Anything retrieved can appear in the answer.

Evaluate the halves separately. Build a set of real queries with labeled relevant passages; 30 to 100 is enough to start. Retrieval metrics: recall@k (is an answer-bearing passage among the k the generator sees?), MRR and NDCG. Generation metrics given gold context: faithfulness (claims supported by the context), answer correctness and citation accuracy. Then measure end to end. When the end-to-end score drops, the split tells you whether to fix chunking, the retriever, the reranker or the prompt.

Common misconceptions

• "Embeddings make keyword search obsolete." Lexical search wins on identifiers, rare terms and new domains; hybrid often beats either alone.
• "A large context window removes the need for retrieval." Cost, latency and uneven use of long contexts still favor a focused set of passages.
• "Similarity scores are relevance probabilities." They are model-specific, uncalibrated and not comparable across models or queries; set thresholds empirically.
• "Reading a few answers is enough evaluation." Without labeled retrieval and faithfulness checks you can neither locate failures nor catch regressions.
• "The model will respect permissions if instructed." Filter before retrieval; a prompt is not an access-control layer.
• "A citation proves grounding." Models can cite passages that do not support the claim; verify support.

How to approach it

Build a small end-to-end pipeline on documents you know, with BM25, a bi-encoder, fusion and a cross-encoder, plus a hand-labeled query set, and measure each stage. Read the IR book's BM25 and evaluation sections and the DPR paper for dense retrieval. In design interviews, walk through ingestion, indexing, retrieval, reranking, generation with citations, access control and evaluation, naming a metric for each stage.