How RAG Works: Index, Retrieve, Generate Explained

Large language models are trained on a snapshot of the internet and then frozen. They do not know your private documents, they do not know what happened this morning, and they will happily fabricate a confident-sounding citation to paper over the gap. Retrieval-Augmented Generation, or RAG, is the fix that became an industry default: before the model answers, go fetch the relevant facts from a knowledge base you control, hand them to the model in its prompt, and let it generate the answer grounded in that evidence.

The reason RAG is everywhere is that it solves three problems at once. It gives the model knowledge it was never trained on, it lets you update that knowledge by editing a database instead of retraining a model, and it lets the model show its work by pointing at the source it used. Once you understand how RAG works, most of the “AI” features you see in products (chat-with-your-docs, support bots, coding assistants that read your repo) stop being mysterious. They are all variations on the same pipeline.

What RAG actually is

RAG is a pipeline with three stages: index your knowledge so it is searchable, retrieve the parts relevant to a question, and generate an answer using those parts as context. The language model only handles the last stage. The first two are an information-retrieval system you bolt onto it. This is the key mental shift: RAG is not a smarter model, it is a model plus a search engine, wired so that the search engine runs first.

This distinction matters for expectations. RAG will not make a model reason better. It will make a model answer factual questions about data it has never seen, and it will make those answers checkable. If your problem is “the model does not know my company’s HR policy,” RAG fixes it. If your problem is “the model gives up on hard math,” RAG will not.

Stage one: building the index

Before you can retrieve anything, your knowledge base has to be in a form that supports relevance search. Raw text does not. The indexing stage turns documents into searchable vectors, and the choices you make here decide the quality ceiling of everything downstream.

Chunking

Documents are too large to retrieve whole and too long to feed many of into a prompt. So you split them into chunks: typically a few hundred words of contiguous text. Chunking sounds trivial and is not. Split on headings and paragraphs, not arbitrary character counts, so chunks stay semantically coherent. Overlap adjacent chunks by a sentence or two so you do not sever a thought at a boundary. A chunk that spans two unrelated topics will never be the best match for either.

Embeddings

Each chunk is passed through an embedding model: a neural network trained to map text to a vector of a few hundred to a few thousand numbers. The training objective is simple in words and hard in practice: texts with similar meaning should land near each other in vector space. A good embedding model puts “how do I reset my password” close to “I forgot my login,” even though they share almost no words. The embedding turns semantic similarity into geometric distance, which is something a database can search.

Storing the index

The vectors and the original chunk text, plus any metadata (source, title, date, access group), go into a vector database or a vector-capable store like PostgreSQL with pgvector. This index is your retrievable knowledge. Updating knowledge means adding, editing, or deleting chunks here: no model retraining required.

Stage two: retrieval

When a user asks a question, that question is embedded with the same model (the model is the coordinate system, so query and documents must share it) and the system searches the index for the closest vectors. The top handful, usually three to ten chunks, become the retrieved context.

Plain vector search is the baseline and often the weak link. Real retrieval systems layer improvements on top. Hybrid search combines vector similarity (good for meaning) with keyword or BM25 search (good for exact terms like product names and error codes) and merges the rankings. Reranking takes the top fifty cheaply-retrieved candidates and runs a dedicated cross-encoder model that scores each query-document pair for true relevance, then reorders them: slower per query but dramatically more precise. Query expansion rewrites the user’s question into multiple variants or adds synonyms before searching. Production RAG systems almost always use hybrid plus reranking; pure vector search is the starting point, not the finish line.

Stage three: generation

The retrieved chunks are assembled into a prompt. A typical construction tells the model: answer the question using only the following context, and if the context does not contain the answer, say so. The chunks are pasted in below the instruction, the question goes last, and the model generates the answer.

The wording of that instruction is doing real work. Telling the model to use only the context suppresses hallucination but also makes it refuse when the answer is in the context but phrased differently. Telling it to use the context as a hint but rely on its own knowledge invites hallucination back in. Most systems land on a middle instruction and then add a requirement to cite which chunk each claim came from, turning the answer into something auditable.

Context assembly also has a budget. The model’s context window is finite, and retrieved chunks compete for space with the question, the instructions, and any conversation history. Retrieving too much fills the window and dilutes the signal; retrieving too little misses the answer. This tension is why retrieval quality, not model size, is usually the lever that moves RAG accuracy most.

Why grounding helps, and where it stops helping

Grounding works because language models are already good at reading a passage and answering a question about it: that is close to their training distribution. RAG moves the hard part out of the model’s parametric memory and into the prompt, where it is controllable and inspectable. The model is no longer recalling; it is reading.

The limit is that grounding only helps if the right evidence was retrieved. If the retrieval stage never fetched the passage containing the answer, no amount of model intelligence can conjure it correctly. A RAG system is a chain, and retrieval is usually the weakest link. Teams that improve their RAG accuracy almost always do so by improving chunking, adding hybrid search, or adding a reranker: rarely by swapping to a bigger model.

Where RAG breaks

Retrieval misses. The right chunk exists but did not surface because the embedding did not capture the match or the query was phrased unlike the source. This is the number-one failure mode and the entire reason reranking exists.
Chunk boundaries sever facts. An answer that spans two chunks may not be retrieved as a unit. Overlap and semantic chunking reduce this.
Stale or contradictory sources. If the index holds two versions of a policy, the model may answer from the old one. Versioning and freshness metadata matter.
Lost in the middle. When many chunks are retrieved, models pay less attention to the ones in the middle of the context. Ordering the most relevant chunks at the start and end helps.
Grounded hallucination. The model cites a chunk that does not actually support the claim, or over-generalizes from it. Citation requirements and faithfulness checks catch this.

Evaluating RAG

Because RAG has two stages, you evaluate both. Retrieval quality uses context-precision metrics: of the chunks you fetched, how many were actually relevant, and did the relevant ones rank at the top? Generation quality uses faithfulness: is every claim in the answer supported by the retrieved context? Frameworks like RAGAS turn these into numbers by using another language model as a judge. The discipline matters because RAG degrades silently: it always produces an answer, so you cannot tell it is broken without measuring whether the answers are grounded.

Production considerations

In real deployments, RAG inherits the constraints of the data behind it. Access control must be enforced at retrieval time: you cannot retrieve a chunk the user is not allowed to see and then hope the model declines to repeat it. Freshness requires a pipeline that ingests new and updated documents and re-embeds them. Latency is the sum of embedding the query, searching, reranking, and generating, so each stage is a budget line. And cost scales with index size and query volume, which is why approximate search and quantized embeddings exist: topics that lead straight into how vector databases work.

Pro Tips
Fix retrieval before the model. When a RAG system gives wrong answers, nine times in ten the right chunk was never retrieved. Before blaming the model, inspect what came back for a failing query. Add a reranker, tune chunk size, or try hybrid search before you touch the generator.
Keep the embedding model fixed across query and index. Query and documents must use the same embedding model and version, or they live in different coordinate systems and similarity is meaningless. Re-embed the whole index if you ever switch models.
Require citations and test for faithfulness. Make the model cite which chunk supports each claim, and run a faithfulness check on a sample of answers. Grounding only counts if the citations are honest.