Ch.12
RAG: Reducing Hallucinations with Retrieval
RAG at a glance
Closed-book = closed-notes exam; RAG = open-book with lookup.
For 'What is the refund policy?' closed-book answers from memory only; RAG embeds, Top-k retrieves, augments, then generates with evidence.
Closed-book: memory only · RAG: retrieve then generate
Closed-book
No external search
RAG
Top-k select
Top k similar chunks
Closed-bookNo external docs → hallucination & stale info risk.
RAGStore → embed → Top-k → augment prompt → generate with citations.
What happens to one question
- A user question arrives; we need grounded, up-to-date answers.
- Embed query and chunks for similarity in the same space.
- Closed-book skips retrieval. RAG picks Top-k chunks.
- Augment prompt from template within context budget.
- Generate; log sources, k; monitor hallucinations.
Imagine a company policy chatbot. An employee asks, "How many PTO days roll over this year?" If the model answers with a plausible but outdated rule, trust breaks immediately. That is hallucination: the model sounds confident without a real source.
RAG (retrieval-augmented generation) answers like an open-book exam: find relevant pages first, paste them into the prompt, then write. A plain closed-book LLM relies on training memory only and cannot reliably track every new PDF, news item, or internal database after training; full retraining every week is unrealistic. The usual flow is retrieve → augment (fill blanks) → generate.
In practice you will see chunks (page-sized slices), embeddings (meaning as number coordinates), a vector DB (a fast similarity index), Top-k (keep only the top k matches), cosine similarity (how aligned two texts are in direction), ctx (max text the model can read at once), prompts (instructions + question), and tokens (small text units). budget = ctx−prompt−query is simply room left on the exam sheet after instructions and the question. This chapter connects those ideas with analogies and numbers — not heavy proofs.
Reading the formulas (RAG)
1. Cosine similarity + Top-k — the search heart
Retrieval scores how aligned question q and chunk c are in meaning. That score is cosine similarity:
is the query embedding and is the chunk embedding. Higher values usually mean more related topics (often read near 0–1). Top- keeps only the top chunks — k=5 means five chunks.
Like a video site ranking titles/descriptions in embedding space, RAG finds the closest policy pages. If scores are 0.92, 0.81, 0.55, 0.30, 0.12 and Top-k=3, use the first three. A "travel expense claim" query should align with meals & transport chunks, not a club newsletter. Remember: angle → top k → attach evidence.
2. Chunking — bite-sized pages
You cannot paste whole PDFs every time, so documents are stored as chunks — bite-sized slices. Think of splitting a 1000-page cookbook into recipe cards; for "broth" you pull soup cards only.
Teams tune chunk size (e.g. 200 tokens) and overlap (e.g. 40 tokens) so sentences are not cut in half. If "refund within 7 days" splits badly, 7 days may land in the next chunk and search misses it. 20-token chunks lack context; 800-token chunks can consume the whole budget per slot. Size + overlap is the foundation of retrieval quality.
3. Context budget — one exam sheet
ctx is one exam sheet per call. Instructions (prompt) and the question (query) use space first; only the remainder holds chunks:
With ctx=4096, prompt=512, query=200, the budget is 3384. At chunk size 200, 1200 // 200= 6 chunks fit. With ctx=8192, prompt=1024, query=256, 6912 tokens remain for chunks. Subtraction and // are two steps of the same story.
4. Prompt template — fill the blanks
Teams send a fixed template and fill `{retrieved_chunks}` with Top- evidence only — like an exam sheet that says "Use the passage below only" plus a paste area and the question.
```
Context:
{retrieved_chunks}
Question: {user_query}
Answer:
```
A filled prompt might paste `[chunk1] Refund within 7 days…` and ask `Is international shipping refundable?` Adding a context-only instruction reduces hallucinations. The template is the box where augment happens.
RAG: Grounded answers with retrieval
1. Open-book vs closed-book: why RAG exists
Using an LLM alone is closed-book: like a closed-notes exam, the model answers from what it memorized in training. RAG (retrieval-augmented generation) is closer to an open-book exam. Before writing, it looks up a page-sized slice (chunk) in a document store (library) and pastes it into the prompt.
The usual pipeline is retrieve → augment (fill blanks) → generate. First pick relevant chunks, fill the `{context}` slot in the template, then let the LLM compose the answer.
Suppose someone asks, "How did this year's PTO rules change?" Closed-book may guess from last year's common sense and still sound fluent. RAG finds page 7 of this year's HR PDF and answers with that text as evidence. The difference is not polish — it is which page was read.
2. Embeddings: text as coordinates
Computers do not compare words like "refund" and "return" by meaning directly. Embedding maps each sentence to a list of numbers (a vector) so similar topics land near each other in that space — like cafes clustering on a map.
Search uses cosine similarity (cos): a score for whether two vectors point in the same direction (angle), which tracks topic overlap better than straight-line distance. "Refund" sits close to "returns / chargebacks"; "lunch menu" sits far away. RAG ranks chunks by cos and keeps the most relevant ones.
3. Chunking & Top- search
A 500-page policy PDF cannot be pasted whole into every prompt. Chunking splits long docs into bite-sized pieces (e.g. 200 tokens) stored as chunks. Smaller slices mean more pieces to search, but slices that are too short lose context.
When a question arrives, embeddings and cos rank chunks; Top- passes only the top k pages to the LLM (k=3 → three chunks). A vector DB acts like a library card catalog — you do not reread the entire PDF each time, you pull similar slices quickly.
If k is too large, irrelevant paragraphs slip in and cost rises. More is not always better — tuning k matters.
4. Context budget & prompt template
Models have a context limit (ctx) — one exam sheet of text per call. Instructions (prompt) and the user question (query) use space first; only the remaining room holds retrieved chunks. That remainder is the chunk budget: budget = ctx−prompt−query.
Teams use a prompt template: a fixed form with `{context}` for retrieved text and `{question}` for the user. Adding "answer only from the context below" discourages guessing outside the pasted pages.
With ctx=4096, prompt=512, query=200, the budget is 3384. At 200 tokens per chunk, roughly 3384 // 200 ≈ 16 chunks fit. Subtraction (budget) and // (max chunks) are two steps of the same story.
Why it matters
1. Why RAG exists — from confident guesses to checkable answers
LLMs are strong at fluent text. On facts they never saw — internal policies, product manuals, yesterday's news — they may still sound sure while being wrong. In business, a wrong answer can reach approvals or customers before anyone notices.
RAG searches first, then writes. You can trace "HR policy PDF, page 12, paragraph 3" and fix documents or search when something drifts. The goal is not zero hallucination, but answers that are much easier to verify.
Example: "Is international shipping covered by the 7-day refund?" — closed-book may generalize domestic rules; RAG pulls shipping clauses and answers conditionally.
2. Keep knowledge up to date without retraining the whole model
A new policy PDF on Monday morning should not force a full 7B retrain. In RAG you chunk → embed → refresh the vector DB; the same LLM can cite the new text the same day.
Analogy: when a textbook gets a new edition, you swap library shelves, not re-teach every student from scratch. Weights stay; which pages are read stays current.
Example: "2026 benefits FAQ" — scenario problems often prefer re-chunk & refresh DB over full retrain for this reason.
3. A smart LLM with the wrong page still fails — retrieval is half the product
Even a strong generator fails when retrieval picks the wrong page. A fluent answer built from a cafeteria menu chunk for a returns question is still a business failure.
When answers drift, check Top-k, chunk size, overlap, and re-rank before chasing temperature (answer randomness). recall@k asks whether the right document landed in the top k — a search report card separate from polished prose.
RAG exists to run evidence, freshness, and search quality as one pipeline, not three unrelated knobs.
How it is used
Step 1: Build the library (ingest & chunk)
Every RAG product starts by collecting sources of truth — policies, manuals, FAQs, notices. Long files are split into chunks, with metadata such as file name and page so users can later ask for citations.
Each chunk is embedded into a vector DB, like building a library card catalog. After that, you do not paste a 500-page PDF into every prompt — you pull relevant slices.
Step 2: Find matching pages (retrieve)
When a question arrives, embed it the same way, rank chunks by cos(q,c), and keep Top- (e.g. 5 pages). If raw scores are weak, add re-rank or deduplication.
If "search works but answers are weird," inspect k, chunk size, and overlap here before tuning generation settings.
Step 3: Assemble the exam sheet (augment)
Selected chunks must fit the chunk budget = ctx − prompt − query. Fill `{context}` in a template and add "answer only from the context below." This augment step finishes the exam sheet so the model reads pasted pages, not empty air.
Step 4: Write the answer (generate) — the full pipeline
The LLM writes the reply; good products show links, pages, and chunk IDs. Log k, chunk size, budget, and scores to tune search when hallucinations or stale answers increase.
End to end: ingest/chunk stores pizza-slice pages (splitter, vector DB); retrieve bookmarks the best matches with embedding, cos, Top-; augment pastes open-book references within the template and budget; generate lets the LLM write from those references. In one line: build library → find pages → paste into template → generate.
Summary
In one sentence, RAG is open-book answering: find relevant pages, paste them into the prompt, then let the model write. The steps are retrieve → augment (fill the template) → generate.
Retrieval picks chunks that match the user question. Cosine similarity cos(q,c) scores how aligned the query q and chunk c are in meaning (direction, not distance); you keep only the Top- best matches. Augmentation fills the `{context}` slot in a prompt template. The model can read at most ctx tokens at once; after instructions (prompt) and the question (query), the room left is the budget: budget = ctx−prompt−query. If each chunk is about 200 tokens, you can fit roughly budget // chunk_size chunks.
For example, when someone asks "What is the refund policy?", you use cos to select three policy pages, not the whole PDF. With ctx=4096, prompt=512, and query=200, you paste evidence only into the space after subtracting 512+200 — not everything you retrieved.
When one search is not enough (questions that need connecting several documents), teams extend RAG with multi-hop retrieval (search again using what you found) or agents (the system plans the next lookup).
Notes for problem solving
For practice problems, start with closed-notes exam (closed-book) vs open-book lookup (RAG). Closed-book has no retrieval; RAG finds pages, fills a template with cos(q,c), Top-, budget, then generates.
Remember retrieve → augment → generate. Numeric items use budget = ctx−prompt−query, then budget // chunk_size for max chunks. temperature changes answer randomness, not search quality.
Common calculations: ctx=4096, prompt=512, query=200 → 3384 · Top-=5 → 5 chunks · budget=1200, size=200 → 1200 // 200= 6. Cosine: "returns" ↔ "refund policy" = similar direction; "cafeteria" = far.
Bank-style samples below.
Example (concept · concept) — closest to RAG?
② retrieve → augment → generate → answer 2
Example (true/false · ox) — RAG uses embedding search → 1
Example (true/false · ox) — closed-book Top-k on PDFs → 0
Sessions often include context budget (vote), Top-k (vote), and chunk count (aggregate/config). The pattern is: subtract prompt and query from ctx, then // by chunk size.
Example (context budget · vote) — limit 4096, prompt 512, query 200 → chunk budget? → 4096-512-200= 3384
Example (Top-k · vote) — Top-k 5 → how many chunks? → 5
Example (chunk count · aggregate) — budget 1200, chunk size 200 → 1200 // 200= 6
Example (integer divide · config) — 1400 // 200 is closest to → 7
Example (scenario · scenario)
"Urgent: refresh latest policy PDFs in QA. First step?
① full LLM retrain
② re-chunk docs & refresh vector DB
③ remove softmax"
→ answer 2
Example (scenario · scenario)
"Retrieval works but answers are wrong. Check first?
① temperature=0 only
② Top-k, chunk size, re-rank
③ GPU driver"
→ answer 2
Example (concept · concept)
"If Top-k is too large, a common downside is?
① no search
② more noise & cost
③ zero embedding dim"
→ answer 2
Example (concept · concept)
"Embedding is best described as?
① optimizer name
② map text to vectors for similarity search
③ batch norm only"
→ answer 2
Example (true/false · ox)
"A prompt template slots retrieved text into `{context}` and `{question}`."
→ answer 1
Example (true/false · ox)
"Large cosine similarity always means small Euclidean distance (simplified teaching)."
→ cos is about direction → answer 0
Example (pipeline · ensemble)
"Top-k 2, 4 summary sentences per chunk → total sentences? ()"
→ 8
Example (context budget · vote)
"Limit 8192, prompt 1024, query 256 → chunk budget?"
→ 6912
Example (chunk count · config)
"Budget 2400, chunk size 200 → 2400 // 200= 12"