What if you have 10 million documents?

The vector database needs to handle scale. Postgres with pgvector works up to about 10 million chunks. Beyond that, look at Qdrant, Pinecone, or Turbopuffer.

How do you handle permissions?

Some users should not see HR documents. Solution: tag each chunk with allowed user groups. Filter at search time, not after. Never let the LLM decide what to show.

What about hallucinations?

Three defenses: (1) give more context, (2) tell the LLM to say "I don't know" when unsure, (3) have a second LLM check the answer against the context.

What if the LLM provider goes down?

Have a backup. Many teams use Claude as primary and GPT as fallback, or the other way around.

How do you handle questions that need information from many documents?

Two-step: first ask the LLM to extract relevant facts from each chunk, then ask it to combine those facts into one answer.

RAG Explained: How to Build an AI That Answers Questions From Your Documents

$ ai-engineer-path

AI Engineering Support

AI Engineering · Beginner guide · 12 min read

← Back to AI Engineering

If this helped, buy me a coffee

Everything is free. Tips keep me writing the rest.

Buy me a coffee →

$ ai-engineer-path

Practical, hands-on guides for backend engineers learning to build with AI.

Topics

AI Engineering

Site

Support FAQ About Contact Glossary

Legal

Privacy Terms RSS feed