Retrieval-Augmented Generation (RAG)

Most "RAG systems that don't work" fail at retrieval, not generation. The model is fine. The chunks you handed it were wrong, stale, or missing the one that mattered. RAG is a retrieval problem wearing an LLM hat.

The pattern: take a user query, fetch the top-K most relevant chunks from an external corpus, stuff them into the prompt as context, let the model answer constrained to what you fetched. Coined by Lewis et al. at Facebook AI Research in 2020 ("Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks"). By 2024, the default integration pattern for production LLM systems that need verifiable, citable, private-corpus answers.

The core loop

query → embed → vector search top-K → stuff into prompt → generate → (cite)

That's it. A working prototype is ~50 lines of Python with numpy dot-products over a few hundred OpenAI embeddings — no vector database required. The scaling concerns (Pinecone, Weaviate, pgvector, FAISS) only start mattering above ~1M chunks or when latency budgets drop below 100ms.

Fine-tune vs. RAG

The decision people get wrong most often.

Concern	Fine-tune	RAG
Fresh data	Re-train, hours-days	Add to index, seconds
Cost	$$$ per training run	$ per query
Provenance	Baked into weights, opaque	Citable to specific chunks
Hallucination	Hard to remove once trained in	Tunable via prompt constraints
Multi-tenant	One model per tenant if isolated	One model, per-tenant retrieval
New reasoning ability	Real gain	None

Fine-tune for new skills or behaviors. RAG for new knowledge. Most teams reach for fine-tuning when they should be fixing chunking.

Where retrieval breaks

Six failure modes account for the vast majority of bad RAG systems I have seen or built:

Chunking that ignores structure. Splitting on every 500 tokens cuts tables in half, separates a heading from its body, and produces chunks that mean nothing standalone. The fix: chunk on semantic boundaries (markdown headers, paragraph breaks) and keep the heading attached.
Domain-mismatched embeddings. Using a general-web embedding model (OpenAI's text-embedding-3-small, Cohere's embed-english) on a legal or medical corpus. Cosine similarity ranks generic phrasing above the term-of-art chunk that actually answers the question. Domain-tuned embedders (Voyage's legal model, BGE-M3, instructor-xl) close most of this gap.
No reranking. Top-K vector similarity is not top-K relevance. A cross-encoder reranker (Cohere Rerank, BGE-reranker) re-scores the top-50 against the query and keeps the top-5. Almost always worth the latency.
Single-pass retrieval on multi-hop questions. "Which of our 2024 contracts had termination clauses triggered by the Texas freeze?" needs the contract list, then the clause lookup, then a date filter. Naive RAG retrieves locally and misses the join.
No metadata filtering. Searching all of a corporate wiki when the question is scoped to one team. Pre-filter on tags, dates, authors. Hybrid search (BM25 + vector) catches the keyword matches embeddings smooth over.
Stale index. The corpus changes; the embeddings don't. Citations become lies. Every production system needs a reindex pipeline.

Where RAG doesn't help

Math and computation. The model still does the arithmetic; retrieving the wrong context can make it worse. Use code-interpreter or function-calling instead.
Highly compositional reasoning. Questions requiring five facts joined across the corpus. Naive RAG retrieves locally. GraphRAG (Microsoft, 2024) and agentic retrieve-loops are the partial fixes.
Voice and style. RAG injects knowledge, not tone. For voice, few-shot or fine-tune.

What's contested

Whether RAG survives the long-context era. Gemini 1.5 Pro handles 2M tokens. Claude 3.5 handles 200K. If you can stuff the entire corpus into the prompt, do you still need retrieval? The empirical answer as of 2026 is yes — long-context models still show "needle in haystack" degradation past ~64K tokens, and the cost-per-query of a 1M-token prompt is roughly 100× a RAG query against a vector store. But the gap is closing, and a class of small-corpus problems where RAG was the obvious answer in 2023 is migrating to long-context-only in 2026.

The second contested question: do you need a vector database at all? A growing camp argues that BM25 + a reranker beats vector search on most enterprise corpora, and that the vector DB industry is a solution looking for a problem. The honest answer is workload-dependent: vectors win on semantic-paraphrase queries; BM25 wins on exact-term and code search.

The wiki as a RAG variant

This wiki is itself a curated RAG store. Instead of dumping raw sources into a vector index, an editorial agent pre-processes material into structured pages with tags, summaries, and cross-realm links. Retrieval becomes filesystem traversal driven by the index. Faster, more accurate, no embedding model — but it caps at thousands of pages, not terabytes. The trade-off is the concept karpathy llm wiki thesis: curation is the bottleneck, not search.

Why this has to do with other realms

Retrieval cost is increasingly the dominant cost of serving an LLM application, not generation. A 5K-token RAG prompt to GPT-4o is cheap; the embedding lookup, reranker, and metadata filter that produced it touch memory and storage hardware in ways the model never does. This makes RAG one of the first AI workloads where the hardware story (see tech neuromorphic computing on retrieval-dominated serving) and the algorithm story have to be designed together. The same pattern shows up in biological memory: the hippocampus is essentially a retrieval index over the cortex, and the energy budget of recall dominates the energy budget of "thinking" in most cognitive tasks.

An open question

If the long-context window keeps doubling every 18 months and inference price keeps halving, at what corpus size does retrieval stop earning its keep? And what new pattern replaces it — agentic search, structured memory, something we haven't named yet?

Key sources

Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (2020, NeurIPS) — the original paper, still the cleanest framing of the architecture.
Microsoft Research, "From Local to Global: A GraphRAG Approach to Query-Focused Summarization" (2024) — the canonical multi-hop fix.
To verify: Anthropic engineering posts on contextual retrieval (2024) — describes the prepend-context-then-embed trick that improved retrieval accuracy ~35% on internal benchmarks.
Pinecone and Weaviate engineering blogs — best practical writing on chunking, reranking, hybrid search in production.

Abhishek's take

I watch this every season when the tools I wrote have to answer "Why did we drop Vendor X’s embroidered kurta set from the Diwali range?" The first version failed because the chunks split the costing sheet from the quality-control photos—so the system cited lead times (irrelevant) instead of the stitching defect notes (critical). Now we pre-attach the QC tag to every chunk tied to a style number, and rerank on recency. Still breaks when the question spans two collections, but that’s a multi-hop problem no one’s solved cleanly on a buying floor. The textbook calls it "retrieval"; we call it "not lying to the merchant about why their order got cut."

Abhishek