There's a moment every developer hits when building with LLMs: the model is impressively fluent but confidently wrong about your stuff. It doesn't know your product docs, your internal wiki, or that policy you updated last week. Two bad instincts follow. The first is to fine-tune a model on your data — expensive, slow to update, and overkill for most cases. The second is to paste everything into the prompt — which hits context limits, costs a fortune per call, and buries the relevant sentence in noise.
Retrieval-Augmented Generation (RAG) is the pragmatic middle path. Instead of teaching the model your knowledge or dumping all of it every time, you fetch only the relevant pieces at question time and hand them to the model alongside the question. The model stays general; your data stays external and up to date. This post walks the whole pipeline without the jargon.
The core idea in one sentence#
Retrieve the few passages most relevant to the user's question, put them in the prompt as context, and ask the model to answer using that context. That's it. Everything else is making each step reliable.
The pipeline, step by step#
RAG has two phases: an offline indexing phase you run when your content changes, and an online query phase you run per question.
1. Chunk your content#
You can't retrieve a 40-page document usefully — it's too big to be "relevant" as a unit. So you split source content into chunks: sections, paragraphs, or fixed-size windows. Chunking is quietly the step that makes or breaks quality.
- Too large, and each chunk carries irrelevant text that dilutes the signal and wastes tokens.
- Too small, and you shred the context a passage needs to make sense.
- A common starting point is a few hundred tokens per chunk with a little overlap between neighbors, so a thought split across a boundary isn't lost.
Prefer splitting on natural structure — headings, paragraphs — over blind character counts when the source has structure to exploit.
2. Embed the chunks#
An embedding turns a chunk of text into a vector — a list of numbers that captures its meaning. The useful property: texts with similar meaning land near each other in vector space, even when they share no exact words. "How do I reset my password" and "steps to recover account access" end up close together.
You run each chunk through an embedding model and store the resulting vector alongside the original text.
3. Store the vectors#
Those vectors go into a vector store — a database that can answer "which vectors are nearest to this one?" quickly. This is where you'll see dedicated vector databases, but plenty of familiar databases now have vector search built in, so you often don't need a new piece of infrastructure just to start.
4. Retrieve at query time#
When a question arrives, you embed the question with the same embedding model, then ask the vector store for the top-k nearest chunks. Those are your candidate passages — the parts of your knowledge base most likely to contain the answer.
5. Generate the answer#
Finally, you build a prompt that includes the retrieved chunks as context and the user's question, with an instruction to answer from the provided context. Roughly:
Use only the context below to answer the question.
If the answer isn't in the context, say you don't know.
Context:
{retrieved chunks}
Question:
{user question}
The model now answers grounded in your actual content rather than its training-time guesses.
Why this beats the alternatives#
- Cheaper than fine-tuning, and updatable instantly. Change a document, re-index that document, done. No training run, no waiting.
- Cheaper than max-context prompting. You send a handful of relevant chunks, not your entire corpus, on every call.
- Traceable. Because you know which chunks you retrieved, you can cite sources and show users where an answer came from — which is often the difference between a toy and something people trust.
Where RAG quietly goes wrong#
RAG is simple to stand up and easy to make mediocre. The failures cluster in a few places:
Retrieval is the ceiling. If the right chunk isn't retrieved, no amount of clever prompting saves the answer — the model simply never sees it. Most "the AI is wrong" complaints in a RAG system are actually retrieval problems. Measure retrieval quality directly before blaming the model.
Chunking mismatches the questions. If users ask broad questions but you chunked into tiny fragments (or vice versa), relevance suffers. Let the shape of real questions inform your chunk size.
No grounding guardrail. Without an explicit "answer only from the context, and admit when you don't know" instruction, the model will happily fill gaps with plausible fiction. That instruction is doing real work — keep it.
Stale index. RAG is only as current as your last indexing run. Wire re-indexing to your content updates, or you'll confidently serve last month's answer.
When you don't need RAG#
Not every AI feature needs a retrieval layer. If the knowledge is small and stable, it may fit directly in a system prompt. If the task is reasoning or transformation rather than recalling facts ("rewrite this," "classify that"), retrieval adds nothing. Reach for RAG when the model needs to know things that are too big for the prompt, change over time, or must be cited. Otherwise you're adding infrastructure for a problem you don't have.
The takeaway#
RAG isn't a model or a library — it's a pattern: chunk, embed, retrieve, generate. Get retrieval right and the rest tends to follow. Start with the simplest version that works, measure whether the right chunks are coming back, and add sophistication only where the numbers tell you to. It's one of the highest-leverage patterns in applied AI precisely because it's so unglamorous.
FAQ#
What does retrieval-augmented generation actually do?#
It fetches the passages from your own data that are most relevant to a user's question and includes them in the prompt, so the model answers grounded in your content instead of only its training data — without any fine-tuning.
Is RAG better than fine-tuning a model?#
For giving a model access to factual, changeable knowledge, usually yes: RAG is cheaper, and you can update the knowledge instantly by re-indexing rather than retraining. Fine-tuning is better suited to teaching a model a style, format, or behavior rather than facts. Many production systems use retrieval as the default and reserve fine-tuning for narrow needs.
What is an embedding in the context of RAG?#
An embedding is a numeric vector that represents the meaning of a piece of text. Texts with similar meaning produce vectors that are close together, which is what lets a vector store find passages relevant to a question even when they use different words.
Why does my RAG app give wrong answers even with a good model?#
Most often the problem is retrieval, not the model: if the chunk containing the answer isn't retrieved, the model never sees it and fills the gap. Check retrieval quality first — tune your chunking and top-k — and add an explicit instruction to answer only from the provided context and to say when it doesn't know.
References: