RAG from Zero · Episode 1

What is RAG? Why AI never read your docs

Your company's docs were never in the model's training data. RAG fixes that without training anything: search first, then ask. We open the exact message our demo bot sends and run the same question with and without RAG.

6:28 videoragretrievalfine-tuningpromptWatch the episode →

The bug: a line from nowhere

We gave our RAG bot, DocBot, every doc of a demo company called Tarnik. A customer asked: how many days do I have to ask for a refund?

DocBot answered: 15 days. Correct, with a source. Then it added one more line on its own:

An older policy allowed 30 days. If your purchase falls under that version, the 30-day window applies.

Where did that line come from? To find the culprit, we need to understand RAG from zero. We find it at the end.

What RAG is: retrieve, augment, generate

RAG = Retrieval Augmented Generation. Three words, three steps:

  1. Retrieve. When a question arrives, our code searches the documents and pulls out the most useful paragraphs (DocBot takes the top 4).
  2. Augment. We send those 4 paragraphs together with the question, all in one prompt.
  3. Generate. The model writes the answer from those paragraphs, not from memory.

Notice something: the model didn’t search anything. Our code did. The model doesn’t even know “RAG” exists. It just reads a longer message.

RAG = search first, then ask. Training needed: zero.

The name comes from Lewis et al., 2020, a paper by researchers at Facebook AI.

Why doesn’t the model know your docs?

ChatGPT knows black holes and Python. So why not your refund policy?

Think of the model as a compiled binary. Everything it learned was compiled in once, months before you typed your first word. Your refund policy was never in the source code, so it’s not in the binary.

So when you ask, it does the only thing it can: it guesses what refund policies usually say. That isn’t lying. It’s autocomplete with confidence.

Why not fine-tune on your docs?

The obvious next idea: compile your docs in. Train the model on the company handbook.

That’s fine-tuning, and it’s like recompiling your whole app every time one database row changes. Policies change every month, prices every week. You’d be retraining forever.

And the part most people don’t know: fine-tuning is good at changing a model’s style (how it talks, what format it writes in), but it’s a weak way to teach new facts. Ovadia et al. (2023) compared the two and found retrieval beat fine-tuning for injecting new knowledge.

Don’t change the binary. Change the input.

Why not paste the whole handbook?

The model has one input: one message. Your instructions, the chat history, everything the model ever sees goes through that one door.

So why not push the whole handbook through it on every question? For a small handbook, honestly, that works. For a big one, every question pays for every page. That’s why RAG: send only the right 4 paragraphs.

The real message DocBot sends

Theory aside, here is DocBot’s actual request for the refund-window question (saved verbatim in rag-lab/docbot/runs/e1/window_prompt.txt, slips shortened here):

SYSTEM:
You are DocBot, the support assistant for Tarnik. Answer the customer briefly,
using only the slips below. Cite the slip id you used in square brackets,
like [file.md#2]. If the slips don't contain the answer, say you don't know.

USER:
Slips:

[refund_policy_2025.md#2]
## 2. Refund window
You can request a full refund within 30 days of your purchase or renewal date. ...

[refund_policy_2026.md#2]
## 2. Refund window
You can request a full refund within 15 days of your purchase or renewal date. ...

[refund_policy_2026.md#5]
## 5. How to request
Open Settings → Billing → Request refund (fastest), or email billing@... ...

[refund_policy_2025.md#5]
## 5. How to request
Email billing@... or open Settings → Billing → Request refund. ...

Question: How many days do I have to ask for a refund?

Three parts:

  • Instructions at the top.
  • 4 slips that search found. Each slip carries its file name (the slip id) and one paragraph.
  • The customer’s question at the bottom.

Instead of the whole handbook, the model gets 4 slips: about 13× smaller. That’s all RAG is. A search, and a bit of copy-paste.

What pasting everything costs

Token counts with cl100k_base (an approximation), at Claude Haiku’s input price of $0.10 per million tokens and ₹88 per $1:

Prompt Tokens ₹ per question ₹ per day at 10,000 questions
Whole handbook (24 docs) 5,194 ₹0.046 ≈ ₹460
RAG message (4 slips) ~400 ₹0.0035 ≈ ₹35

On one question the difference is invisible. At scale it isn’t. And our demo handbook is tiny; real company docs are many times bigger.

Why Ctrl+F isn’t enough

One question remains: how does the search know which paragraph is right?

The simplest search is Ctrl+F. A customer types refund, and Ctrl+F finds “refund” 58 times in the handbook.

Now the same customer, in their own words: “paisa wapas kab milega?” Ctrl+F searched the whole handbook: zero results. The document says refund; the customer said paisa wapas. Different words, same meaning.

So search has to match meaning, not words. That means turning text into numbers where meaning can be measured. Those are embeddings, the whole next episode. For now, assume our search understands meaning.

Our test: without vs with RAG

Same question, same model (Claude Haiku). Pause and guess first: what does the model say without RAG? And with it?

Without RAG: 10 runs

System prompt: “You are DocBot, the support assistant for Tarnik. Answer the customer briefly.” No documents. Temperature 1, 10 runs.

10 out of 10: “I don’t have Tarnik’s refund policy, so I can’t give you an accurate number of days.”

It never invented a number. Better than guessing, but the customer still has no answer.

With RAG

Search pulled the 4 slips above, the model read them and said:

You have 15 days from your purchase or renewal date to request a full refund [refund_policy_2026.md#2].

Correct, with its source.

Finding the culprit

Look at which slip search put on top:

Rank Slip Cosine similarity
1 refund_policy_2025.md#2 (old policy: 30 days) 0.5650
2 refund_policy_2026.md#2 (current: 15 days) 0.5570
3 refund_policy_2026.md#5 (how to request) 0.4952
4 refund_policy_2025.md#5 (how to request) 0.4890

The stale 2025 policy ranked first. That’s where the 30 days came from.

The model spotted the current policy from the file name (the paragraphs themselves don’t contain the year), so it answered 15. But it still added at the end that the old 30-day window might apply.

The real bug: that old document should never have been in the index. Stale documents are one of the most common RAG bugs, and we fix it later in the course.

Same model, same question. Only the input changed, and the answer became right.

Does RAG stop wrong answers?

No. Stanford researchers (Magesh et al., 2024) tested legal AI tools that use RAG. They still got 17% to 33% of answers wrong.

Because if search brings the wrong paragraph, the model writes its answer from that wrong paragraph. Garbage in, confident garbage out. Building RAG is easy. Running it well is the real work, and that’s what this course is about.

“Claude Code only runs grep.” Is RAG dead?

Someone always comments: Claude Code just runs grep, no vector database. So is RAG over?

No. grep is search too. Claude Code greps first, adds what it found to the message, then answers. Retrieve, augment, generate. That’s RAG.

In code, grep is often enough, because names in code are exact: search for the function name and you land on the right file. But a customer says paisa wapas, not refund, so support bots need search that understands meaning.

And Cursor? It does both: grep and embedding-based semantic search (Cursor’s blog, Nov 2025).

The only difference is who searches:

  • DocBot (classic RAG): our code searches once, then the model answers.
  • An agent (agentic RAG): the model itself decides what to search, as many times as it wants.

Same three steps, a smarter searcher. We race the two later in the course.

When you don’t need RAG at all

If your handbook is small (roughly under 500 pages), you may not need RAG at all. Put the whole thing in the prompt, with prompt caching so you don’t pay full price for it on every question (Anthropic’s Contextual Retrieval post makes the same point). We’ll do that trade-off with real numbers later in the course.

The RAG from Zero map

The RAG recipe is simple:

QUESTION → SEARCH → TOP 4 SLIPS → MODEL → ANSWER

But every box has a whole world inside it, and that’s the map of this course:

  • How does search understand meaning? Embeddings.
  • How do you find the nearest slip among millions in milliseconds? Vector databases.
  • How are slips cut from documents? Chunking.
  • Then put it all together and get a first honest score.
  • When meaning search fails on IDs: keyword search. When the question itself is bad: query rewriting. And when you don’t need vectors at all.

Every episode, DocBot gets a new station, and we score it on the same 50 questions.

Cheat sheet

  • RAG = search first, then ask. Retrieve → augment → generate. No training.
  • The model doesn’t search. Your code does (or, in agentic RAG, the model calls search as a tool).
  • The model is a compiled binary: it only knows what was in its training data.
  • Fine-tuning changes style, not facts. Don’t use it to teach your docs.
  • Pasting everything works for small docs, and gets expensive and noisy as they grow.
  • The whole trick is what goes into one message: instructions + slips (with ids) + question.
  • Ask the model to cite slip ids. It’s how we caught the stale document.
  • RAG is only as good as its search. Wrong slip in, confident wrong answer out.
  • Stale docs in the index are a real bug. Remove old versions.

Next episode: embeddings. The paisa wapas question, and how text becomes numbers that understand meaning.

Sources

  • Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020. arXiv:2005.11401
  • Ovadia et al., Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs, 2023. arXiv:2312.05934
  • Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Stanford, 2024. arXiv:2405.20362
  • Anthropic, Introducing Contextual Retrieval, 2024.
  • Cursor blog, semantic search in the agent, Nov 2025.
  • All DocBot numbers were measured by us on 2026-11-06 with Claude Haiku and text-embedding-3-small.