What you need to know
Retrieval-augmented generation (RAG) answers questions from a body of knowledge too large, too specialized or too changeable to rely on the model's training. The pipeline has two halves:
- Indexing (offline): collect documents, split them into chunks, compute representations, and store them in an index.
- Answering (online): turn the question into a query, retrieve candidate chunks, optionally rerank them, and pass the best ones to Claude with instructions to answer from them and cite.
Remember lesson 1.2: retrieval is for stable knowledge. Live state comes from tools.
Do you need RAG at all?
If the whole knowledge base fits comfortably in the context window (a few hundred pages at most), loading it directly, with prompt caching, can be simpler and more accurate than building a pipeline. RAG earns its complexity when the corpus is large, grows, or changes.
Chunking: by the shape of the documents
The chunk is the unit that comes back. Choose the method by what the documents are, not by a default size.
| Method | How | Fits |
|---|---|---|
| Fixed-size with overlap | Equal spans, overlapping a little | Uniform, unstructured text with weak natural boundaries |
| Semantic | Split where the topic changes | Prose, where each chunk must make sense on its own |
| Structural (hierarchical) | Follow the document's sections and subsections | Manuals, contracts, policies, where the section heading carries meaning |
A common failure: a chunk loses the context that made it meaningful ("the limit is 30 days", but for which policy?). Fixes include keeping section titles and document names with each chunk, and contextual retrieval, which prepends a short, generated description of where the chunk sits in its document before indexing it.
Indexing: by how people ask
| Index | Matches | Fits |
|---|---|---|
| Dense (embeddings) | Meaning, even with different words | Questions phrased unlike the source text |
| Sparse (keyword, such as BM25) | Exact terms: part numbers, codes, names | Questions that hinge on a specific identifier |
| Hybrid | Both, with results merged | Mixed query patterns, which is most production traffic |
Three trade-offs to name
Every retrieval design balances retrieval quality (does the right chunk come back?), latency (how much time retrieval adds) and maintenance (how much it costs to keep the index correct as documents change). Reranking improves quality and adds latency; finer chunking improves precision and increases index size; frequent re-indexing improves freshness and costs compute.
Keeping it fresh
Decide how changes reach the index: re-index changed documents on publish, on a schedule, or both; remove withdrawn documents; record the source version in each chunk so citations point to the right edition.
In the exam
RAG design questions give a corpus and a query pattern, then ask for chunking and indexing choices, or describe a failure (wrong section retrieved, exact codes not found, stale answers) and ask for the fix. Match chunking to document shape, indexing to query pattern (hybrid when identifiers matter), and freshness to how often the documents change. If the corpus is small, consider whether RAG is needed at all.
Check your understanding
Technicians search a manual library using exact fault codes such as "E-4417". Embedding search often returns similar-sounding but wrong codes. What fixes this?
Show answer
Answer: D. Exact identifiers are what keyword (sparse) indexes match well. A hybrid index keeps the semantic matching for other questions and recovers exact codes.
Answers about insurance policies mix up conditions from different sections. The policies are split into fixed 400-token chunks. What is the best fix?
Show answer
Answer: C. Structured documents should be chunked along their structure, with the context that gives each chunk its meaning. Smaller fixed chunks lose even more context.
A team has 120 pages of reference material that rarely changes and is used on every request. Which TWO options are sound?
Show answer
Answer: A, C. A small, stable corpus may fit the context window, where caching makes it cheap. Retrieval should be added only if measurement shows a need.
Apply it: Design Corvane's product knowledge pipeline
Corvane's product-question route needs retrieval over 2,300 product manuals (PDFs with sections such as assembly, care and dimensions) and 400 help-centre articles updated weekly. Customers ask in their own words ("my table wobbles"), and sometimes quote product codes ("CV-2210-OAK").
- Choose chunking for each source, with a reason.
- Choose the index type and justify it with the query patterns.
- Design freshness: how revised manuals and weekly article updates reach the index, and how citations stay correct.
- Define two retrieval metrics you will track.
Done when: each metric can be computed without reading every answer by hand.
Show model answer
1. Chunking
| Source | Chunking | Reason |
|---|---|---|
| Product manuals | Structural: by section, with product name, product code and section path attached | Answers depend on the section ("assembly" vs "care") and on which product |
| Help-centre articles | One chunk per article, split by heading only when long | Articles are short and self-contained |
Contextual descriptions are generated for manual chunks so a chunk like "tighten bolts B and C" carries "Assembly, CV-2210 dining table".
2. Index. Hybrid: dense embeddings for problem descriptions in customers' words, plus keyword matching for product codes and part letters, with results merged and the top few reranked before generation.
3. Freshness. Manuals are re-indexed when the catalogue marks a product revision, removing the old version's chunks; articles are re-indexed on publish each week. Each chunk stores the document ID and version, and answers cite them, so a citation always points to the edition that was used.
4. Metrics. Recall at 5 on a labelled set of 300 real questions (is a correct chunk among the top five?), and the share of answers whose citation points to the right product, measured automatically from the product code in the question and in the cited chunk.