Language models are brilliant at reasoning and language, and unreliable at facts. They cannot know your internal documents, they go stale the moment training ends, and when they do not know something they tend to invent a confident answer. Retrieval-augmented generation, or RAG, is the technique that fixes this — and it has become the default pattern for building AI applications on your own data.

This guide explains how RAG works, the pieces you need to build it, and where it fits alongside the alternatives. It sits at the intersection of two earlier guides: vector databases provide the retrieval, and language models provide the generation.

What is retrieval-augmented generation?

RAG is a simple idea with outsized impact: instead of relying only on what a model memorized during training, you retrieve relevant information at query time and hand it to the model as context. The model then generates its answer grounded in that supplied material rather than from memory alone.

Think of the difference between a closed-book exam and an open-book one. A bare language model takes the closed-book test from memory. RAG turns it into an open-book exam, where the right pages are placed in front of the model before it answers.

Why RAG matters

  • It grounds answers in real sources. Responses are based on documents you provide, sharply reducing hallucination.
  • It uses current, private knowledge. Your data never needed to be in the training set. Update the source and the answers update too.
  • It is citable. Because you know which documents were retrieved, you can show users where an answer came from.
  • It is cheaper than retraining. Adding knowledge means adding documents, not running an expensive fine-tune every time facts change.

How RAG works, step by step

A RAG system runs in two phases.

Indexing (done ahead of time):

  1. Collect your source material — docs, tickets, wiki pages, PDFs.
  2. Chunk it into passages small enough to be precise but large enough to keep meaning intact.
  3. Embed each chunk into a vector using an embedding model.
  4. Store those vectors in a vector database so they can be searched by similarity.

Retrieval and generation (at query time):

  1. Embed the user's question with the same model.
  2. Search the vector database for the most similar chunks.
  3. Assemble a prompt that combines the retrieved context with the user's question.
  4. Generate the answer with a language model, instructed to rely on the supplied context.

The quality of a RAG system is decided far more by the retrieval half than by the model. Garbage context produces a confident, wrong answer no matter how capable the model is.

The building blocks

What makes RAG good or bad

Most RAG projects succeed or fail on the details:

  • Chunking strategy. Too large and retrieval is imprecise; too small and context is lost. This is worth tuning carefully.
  • Hybrid search. Combining keyword and vector search catches both exact terms and semantic matches.
  • Re-ranking. A second pass that reorders retrieved chunks by relevance often improves answers more than swapping the model.
  • Evaluation. Measure retrieval quality and answer faithfulness, not just vibes. Tools like Weights & Biases help track this rigorously.

RAG vs. fine-tuning vs. long context

These are often framed as competitors; they solve different problems.

  • RAG is for knowledge — facts that change, are private, or are too large to fit in a prompt. It is the right default for document Q&A and assistants.
  • Fine-tuning is for behavior — teaching a model a tone, format, or skill. It does not reliably add factual knowledge. We compare the two in depth in fine-tuning vs. RAG.
  • Long-context prompting — pasting everything into a large context window — works for small, one-off document sets but is costly and does not scale to large or frequently changing corpora.

In practice these combine: a fine-tuned model for style, RAG for knowledge, all orchestrated together.

Frequently asked questions

Does RAG eliminate hallucination? It reduces it substantially but does not eliminate it. A model can still misread or over-extend retrieved context. Good prompting, re-ranking, and showing sources all help.

Do I need a vector database for RAG? For anything beyond a tiny prototype, yes — it is the component that makes retrieval fast and scalable. See our guide to vector databases for the options.

Can I build RAG with open-source models? Absolutely. RAG is model-agnostic. Many teams pair an open-source model with a self-hosted vector database to keep the entire pipeline private.

Is RAG still relevant as context windows get bigger? Yes. Even with huge context windows, retrieval is more accurate, far cheaper, and more scalable than stuffing everything into every prompt. Large contexts complement RAG rather than replacing it.

Build your stack

The components of a RAG pipeline — embedding providers, vector databases, models, and orchestration tools like LangChain — are all in the ProductListo directory. For the bigger picture, see the best AI tools for 2026.

Have a RAG or retrieval tool we should list? Submit it to ProductListo.