RAG Explained: How to Build an AI That Answers Questions From Your Documents

If you build with AI as a backend engineer, you will hit this problem early. It is the first non-trivial thing most teams ship with LLMs.

“Design a system that lets users ask questions about a large set of documents.”

The answer is called RAG, short for Retrieval Augmented Generation. This guide explains what it is, how it works, and how to build one, with no jargon and no assumed knowledge.

What is RAG?

LLMs like ChatGPT and Claude have two problems that block most useful work:

1. They do not know about your documents: your company wiki, product manuals, support tickets.
2. They sometimes make things up. That is called hallucination.

RAG fixes both with one idea: find the right documents first, hand them to the model as context, then let the model answer from that context instead of from memory.

Here is the difference:

flowchart LR
    subgraph WO ["Without RAG"]
        direction LR
        A1[User question] --> B1[LLM]
        B1 --> C1[Answer from training data<br/>May be outdated or wrong]
    end

flowchart LR
    subgraph W ["With RAG"]
        direction LR
        A2[User question] --> S[Search your documents]
        S --> D[Found 5 relevant chunks]
        D --> B2[LLM]
        A2 --> B2
        B2 --> C2[Answer from YOUR documents<br/>With citations]
    end

Think of it like an open-book exam. Without RAG, the AI takes a closed-book test, relying only on what it learned during training. With RAG, the AI gets to look up the answer in your textbook before responding.

A real example

Imagine you work at a software company. Your support team gets hundreds of questions a day:

“How do I reset my password if I never set a recovery email?”

Your company has a knowledge base with 500,000 help articles. A RAG system would:

1. Take the question
2. Search the knowledge base for relevant articles
3. Find the top 5 articles about password resets
4. Send those articles plus the question to the LLM
5. Get back an accurate answer that cites the specific articles

The user gets a correct, sourced answer. The support team stops re-typing the same reply.

The big picture

A RAG system has two main pipelines that run at different times.

The setup pipeline runs in the background. It prepares your documents for searching.

flowchart LR
    DOC[Your documents<br/>PDFs, web pages] --> PARSE[Read the text]
    PARSE --> CHUNK[Split into<br/>small pieces]
    CHUNK --> EMBED[Turn each piece<br/>into numbers]
    EMBED --> STORE[(Save in a<br/>vector database)]

The query pipeline runs every time someone asks a question.

flowchart LR
    Q[User question] --> SEARCH[Search the database]
    SEARCH --> TOP[Get the top 5 matches]
    TOP --> LLM[Send to the LLM<br/>with the question]
    LLM --> ANS[Answer the user]

Each step has its own failure modes. Walk through them in order.

Step 1: Read the documents

Your documents come in many shapes: PDFs, web pages, Word docs, Slack messages, even images.

You need to turn all of them into clean text. This sounds simple. It is not.

• HTML pages have menus, ads, and footers that should be removed.
• PDFs are designed for printing, not reading. Tables and columns get mangled.
• Images and diagrams need a special model to extract text from them.

Start simple. Use one library per file type. Do not try to handle every weird case in the first version; the long tail of malformed inputs will eat weeks if you let it.

Useful tools:

• trafilatura for cleaning HTML
• pdfplumber for basic PDFs
• A vision model (Claude, GPT-4o) for tricky PDFs with images and tables

Step 2: Split into chunks

Once you have clean text, you split it into smaller pieces called chunks.

Why? Two reasons:

1. LLMs have a limited memory (called the context window). You cannot fit a 200-page manual into one prompt.
2. You want to find the exact paragraph that answers a question, not the whole document.

flowchart LR
    DOC[One long document<br/>50,000 words] --> SPLIT[Split by sections<br/>and paragraphs]
    SPLIT --> C1[Chunk 1<br/>~400 words]
    SPLIT --> C2[Chunk 2<br/>~400 words]
    SPLIT --> C3[Chunk 3<br/>~400 words]
    SPLIT --> CN[...and so on]

A good chunk:

• Is 400 to 800 words long
• Stops at natural breaks (end of a paragraph or section), not mid-sentence
• Has a little overlap with the previous chunk so context is not lost

A bad chunk:

• Cuts off in the middle of a table
• Splits a code block in half
• Is too long (the AI cannot focus) or too short (no context)

Step 3: Turn chunks into numbers (embeddings)

Computers cannot search text the way we do. They need numbers.

An embedding is a list of numbers that represents the meaning of a piece of text. Two chunks with similar meaning will have similar numbers. Two chunks with different meanings will have different numbers.

flowchart LR
    T1["How do I reset<br/>my password?"] --> M[Embedding<br/>Model]
    T2["I forgot my login<br/>credentials"] --> M
    T3["What is the<br/>weather today?"] --> M
    M --> N1["[0.12, -0.45, 0.78, ...]"]
    M --> N2["[0.14, -0.42, 0.81, ...]"]
    M --> N3["[-0.88, 0.55, -0.23, ...]"]

The first two get similar numbers because they mean similar things. The third gets very different numbers because it is about a different topic.

You use a pre-trained embedding model (a small AI just for this job). Popular ones in 2026:

• text-embedding-3-large (OpenAI), easy, paid
• voyage-3 (Voyage AI), high quality, paid
• nomic-embed-text-v2 (open source, free, run it yourself)

You save these embeddings in a special database called a vector database.

Step 4: Find the right chunks for a question

When a user asks a question, you:

1. Turn the question into an embedding (using the same model)
2. Search the database for chunks with similar embeddings
3. Get the top 5 or 10 results

flowchart LR
    Q[User asks:<br/>How do I reset<br/>my password?] --> QE[Question<br/>embedding]
    QE --> DB[(Vector<br/>Database)]
    DB --> R1[Chunk: To reset<br/>your password...]
    DB --> R2[Chunk: If you forgot<br/>your login...]
    DB --> R3[Chunk: Account<br/>recovery steps...]

This is called semantic search. It finds chunks based on meaning, not just matching words.

Semantic search alone is not enough. It is good at meaning, bad at exact terms: product names, error codes, SKUs, function names. For those, keyword search (BM25) is better. Run both, fuse the results, and you get hybrid search, the standard for production RAG. Covered in article 06.

Step 5: Re-ranking (the cheap accuracy win)

Searching gives you 20-50 candidate chunks. But the order is not perfect. The truly best chunk might be ranked seventh, not first.

A re-ranker is a small model that looks at the question and each candidate chunk, then scores how relevant each one is.

flowchart LR
    SEARCH[Search returns<br/>top 50 chunks] --> RERANK[Re-ranker<br/>scores each one]
    RERANK --> TOP5[Best 5 chunks<br/>go to the LLM]

Re-ranking improves accuracy by 10-15% for very little cost. It is the cheapest accuracy win in production RAG. Always add it.

Step 6: Send everything to the LLM

Finally, you give the LLM three things:

1. A system prompt telling it how to behave
2. The top chunks (as context)
3. The user’s question

You are a helpful assistant. Answer the question using ONLY
the context below. If the context does not contain the answer,
say "I do not know." Always cite which chunk you used.

CONTEXT:
[Chunk 1] To reset your password, go to Settings...
[Chunk 2] If you do not have a recovery email, contact support...
[Chunk 3] Account recovery requires...

QUESTION: How do I reset my password if I never set a recovery email?

The LLM reads the context and writes an answer based on it, with citations like [Chunk 2].

The full picture

Here is everything together:

flowchart TB
    Q[User question] --> EMB[Convert to embedding]
    EMB --> HYBRID[Hybrid search<br/>semantic + keyword]
    HYBRID --> RR[Re-rank candidates]
    RR --> CTX[Top 5 chunks]
    CTX --> PROMPT[Build prompt:<br/>system + context + question]
    Q --> PROMPT
    PROMPT --> LLM[LLM generates answer]
    LLM --> CITE[Answer with citations]

That is RAG. Six steps, every one of them a place where things go wrong in production.

Common beginner mistakes

Mistake 1: Using huge chunks. A 5,000-word chunk is too much. The LLM cannot focus. Stick to 400-800 words.

Mistake 2: Only using semantic search. Pure vector search misses exact matches like product names. Always combine with keyword search.

Mistake 3: Forgetting about evaluation. How do you know your RAG works? Build a test set of 50 question/answer pairs and measure how often the right chunk is found. We cover this in article 08: LLM Evaluation Pipelines.

Mistake 4: Treating documents as already clean. Bad parsing causes the worst kind of RAG failure: the right chunk is found, but the text inside it is garbled. Always check parsed output before chunking.

Mistake 5: No “I don’t know” option. If the context does not contain the answer, the LLM should refuse, not guess. Tell it so in the prompt.

Questions you will face in production

“What if you have 10 million documents?” The vector database needs to handle scale. Postgres with pgvector works up to about 10 million chunks. Beyond that, look at Qdrant, Pinecone, or Turbopuffer.

“How do you handle permissions?” Some users should not see HR documents. Solution: tag each chunk with allowed user groups. Filter at search time, not after. Never let the LLM decide what to show.

“What about hallucinations?” Three defenses: (1) give more context, (2) tell the LLM to say “I don’t know” when unsure, (3) have a second LLM check the answer against the context.

“What if the LLM provider goes down?” Have a backup. Many teams use Claude as primary and GPT as fallback, or the other way around.

“How do you handle questions that need information from many documents?” Two-step: first ask the LLM to extract relevant facts from each chunk, then ask it to combine those facts into one answer.

What to remember

• RAG = Find documents + Show them to the LLM + Get a grounded answer
• Two pipelines: setup (offline) and query (online)
• Key parts: parse, chunk, embed, store, search, re-rank, generate
• Hybrid search beats pure vector search
• Re-ranking is the cheapest accuracy win
• Always have an “I don’t know” option

What to study next

If you want to go deeper:

1. Anthropic’s contextual retrieval blog post, explains a clever trick to make chunks more searchable
2. The RAGAS library docs, shows how to evaluate RAG systems
3. Hamel Husain’s writeups, practical tips from someone who builds production AI

Then article 08: How to know if your AI system is actually working, the evaluation pipeline that tells you whether any of this is working in production.

Further reading

• Lewis et al. (2020) — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. The original RAG paper. Worth reading for the framing.
• Anthropic — Contextual retrieval. Modern improvement to RAG that prepends chunk context before embedding. 35-50% improvement in retrieval.
• Pinecone Learning Center. Excellent practical guides on vector search, RAG, retrieval evaluation.
• LangChain — RAG concepts. Implementation patterns and architectural conventions.

Where this article comes from. This is a synthesis of common practice in AI engineering as of 2026, not a citation of any single paper. The sources above are where the broad mechanics and specific numbers come from. If you find an error or have a better source for a claim, the article gets fixed within a day, send me a note.