A pretrained large language model (LLM) knows what was in its training data and nothing about your company's documents. Ask it about your refund policy and it will either say it doesn't know or, worse, invent a plausible one. Retrieval-augmented generation (RAG) fixes this without touching the model: before each answer, you look up the few passages in your own documents that bear on the question, put them in the prompt, and ask the model to answer from them and cite them.

You need it when the documents are private, change often, or are too large to send whole. You don't when the whole collection fits comfortably in the context window and rarely changes, such as a few short policy pages you can simply paste in, or when the problem is the answer's tone or format rather than its facts.

This tutorial covers the parts you design yourself: how to cut documents into chunks, how to retrieve the right ones, how to assemble them into the context, how to make the answer cite its sources or refuse, and how to tell which step broke when an answer is wrong.

We'll use one example throughout. A payroll software company runs a support assistant in a multi-turn chat widget. It answers customer questions from the company's help center: 8k help articles, release notes, and policy pages, about 6M tokens in total, updated weekly. The model has a 128k-token context window. Every answer has to cite the article it came from, and when the docs don't cover a question the assistant has to say "I don't know" and hand off to a person.

RAG is the most common design question in AI engineering interviews, and it often fills most of the round. A mid-level answer explains why retrieval beats pasting documents into the prompt, sketches the pipeline with numbers, picks a chunking strategy, and requires citations. A senior answer also says when to add keyword search, a reranker, or query rewriting and what each costs, and it can diagnose a wrong answer from a logged trace.