RAG: Giving Your AI Real Knowledge

RAG (Retrieval-Augmented Generation) connects an AI model to your own data at inference time, solving the training-cutoff problem without retraining the model. Learn how it works and how to build it.

TL;DR: RAG stands for Retrieval-Augmented Generation. Instead of asking a model to "remember" facts from training, you fetch the right documents at query time and hand them to the model as context. The model reads them, then answers. It is the single most practical pattern for making AI useful on your own data - no retraining required.

The Core Problem RAG Solves

Every large language model has a knowledge cutoff. It learned from a snapshot of text that ended on a specific date, and it cannot update itself after that. Ask it about events last month, your internal product docs, or a customer's order history - and it either guesses or refuses.

Fine-tuning can bake new knowledge in, but it is expensive, slow, and you still cannot update it in real time. You would need to retrain every time your data changes.

RAG sidesteps both problems. The model's weights stay frozen. Instead, at the moment a user asks a question, you pull the relevant documents from an external store and drop them into the model's context window alongside the question. The model reads what it needs, then generates an answer grounded in that text.

That is it. That is the whole idea.

How the Pipeline Works

A RAG system has two distinct phases: an offline indexing phase (you run this once, or whenever data changes) and an online query phase (you run this on every user request).

Phase 1 - Indexing your knowledge base

1. Ingest - collect your source documents (PDFs, markdown files, database rows, transcripts, whatever).
2. Chunk - split each document into smaller pieces. A common starting point is 512 tokens per chunk with a 10-20% overlap between adjacent chunks so you do not lose meaning at boundaries.
3. Embed - run each chunk through an embedding model. The model converts text into a vector - a list of numbers that captures semantic meaning. Chunks about similar topics land near each other in this vector space.
4. Store - write every chunk and its vector into a vector database (Pinecone, pgvector, Weaviate, Chroma, and many others). The database is optimized to find the nearest vectors fast.

Phase 2 - Answering a query

1. Embed the question - run the user's query through the same embedding model to get its vector.
2. Retrieve - search the vector database for the chunks whose vectors are closest to the query vector. Retrieve the top 3-10.
3. Augment - build a prompt that includes the retrieved chunks plus the user's question.
4. Generate - send the augmented prompt to the language model. The model reads the chunks, then writes an answer grounded in them.

A minimal prompt looks like this:

You are a helpful assistant. Use only the context below to answer.

Context:
---
[retrieved chunk 1]
[retrieved chunk 2]
[retrieved chunk 3]
---

Question: {user_question}
Answer:

Why Vectors? A Plain-English Explanation

Classical search matches keywords. If the user types "car" and your document says "automobile," keyword search misses it. Vector search does not care about exact words - it works on meaning.

An embedding model maps every piece of text to a point in a high-dimensional space (think of it as a map where concepts live near related concepts). "Car," "automobile," and "vehicle" all land close together. "Sourdough bread" lands somewhere completely different.

When you search by vector, you are asking: "which of my stored chunks are most similar in meaning to this question?" That is why RAG can answer questions even when the user's phrasing does not match the words in your documents.

OpenAI's Retrieval API documentation describes vector stores as indices where files are "automatically chunked, embedded, and indexed" - semantic search surfaces "results with few or no shared keywords, which classical search techniques might miss."

Contextual Retrieval - Anthropic's Improvement

Standard chunking has a real problem: when a chunk is cut out of a document, it loses the surrounding context. A chunk might say "the policy changed in Q3" but nothing in that chunk tells the retriever which policy, which company, or which year. Embedded in isolation, the chunk becomes hard to find with the right query.

Anthropic published an engineering post in September 2024 introducing Contextual Retrieval to address exactly this. Before embedding each chunk, you prepend a short AI-generated summary that explains what that chunk is about in the context of the whole document. The chunk still contains the original text, but now it also carries just enough context for the embedding to be meaningful.

The results, measured by retrieval failure rates, are significant:

• Contextual Embeddings alone reduce failure rates by 35% (from 5.7% to 3.7%).
• Contextual Embeddings combined with BM25 hybrid search reduce failure rates by 49% (from 5.7% to 2.9%).
• Adding a reranker on top of both cuts failure rates by up to 67% (from 5.7% to 1.9%).

The technique uses Claude itself to generate the chunk context via prompt caching, which keeps the extra cost manageable - Anthropic reports that caching can cut the incremental cost by up to 90%.

Building a RAG System: What You Actually Need

You need four things:

• An embedding model - OpenAI's text-embedding-3-large is a strong default. Open-weight alternatives like BGE-M3 work well for self-hosted setups.
• A vector store - pgvector if you already run Postgres; Pinecone or Weaviate if you want a managed service; Chroma if you want something local and fast to prototype with.
• A language model - any capable model works (Claude, GPT-4o, Llama 3, Gemini). The model just needs to be good at following instructions and reading provided context.
• An orchestration layer - something to glue the pieces together. LangChain and LlamaIndex are popular open-source options. OpenAI's Responses API includes a built-in file search tool backed by managed vector stores - you upload files, and the platform handles chunking, embedding, and retrieval automatically. With the Claude API, you build the pipeline explicitly and can apply techniques like Contextual Retrieval at each step.

Common Mistakes and How to Avoid Them

Chunks that are too big or too small

A 4,000-token chunk retrieves too much noise. A 50-token chunk loses too much meaning. Start at 512 tokens with 10% overlap and tune from there based on your documents' structure.

No reranking

Vector similarity is a good filter, not a perfect judge. A cross-encoder reranker (Cohere Rerank is a popular choice) reads the query and each candidate chunk together and scores them more accurately. Add it after retrieval and before the final prompt.

Retrieving too few chunks - or too many

Fetch 3-5 chunks if your model has a small context window. Fetch 10-20 if you have room and use a reranker to pick the best before sending them. Sending 30 mediocre chunks buries the good answer in noise.

Skipping metadata

Tag every chunk with its source (filename, URL, timestamp, author). It makes citations trivial, and you can filter by metadata before vector search ("only search documents tagged 2026") to sharply improve precision.

Not telling the model what to do when it does not know

Instruct the model explicitly: "If the answer is not in the provided context, say so." Otherwise it will hallucinate from training data - which defeats the entire purpose of RAG.

RAG vs. Fine-Tuning - When to Use Which

• Use RAG when your data changes frequently, when you need cited answers, when you need to search across many documents, or when you want to add knowledge without touching the model.
• Use fine-tuning when you want the model to adopt a consistent style, tone, or format - not to learn new facts. Fine-tuning is about behavior, not knowledge.
• Use both when you need a model that behaves in a domain-specific way AND can answer questions from a live knowledge base.

RAG is almost always the right first move for knowledge-grounding tasks. It is faster to ship, cheaper to maintain, and far easier to update when your data changes.

Key takeaways

• RAG separates knowledge (in a database) from reasoning (in the model) - update each independently.
• The pipeline has two phases: index offline, retrieve + generate online.
• Embedding models convert text to vectors so you can find semantically similar content, not just keyword matches.
• Chunk size, overlap, and metadata quality matter as much as the model you choose.
• Anthropic's Contextual Retrieval cuts retrieval failures by up to 35% (embeddings alone), 49% (+ BM25), and 67% (+ reranker) - each step adds cost but also meaningful accuracy gains.
• Always instruct the model to say "I don't know" when the context does not contain the answer - do not let it drift back to training data.
• Fine-tuning shapes behavior; RAG supplies knowledge. They solve different problems.

Try this next: once you understand how RAG grounds model outputs in your data, the natural next step is understanding how models use tools to take action on that data - not just read it. What Are AI Tools and Function Calling? walks through exactly that.