Hybrid Search: When Pure Vector Search Isn’t Enough

If you read enough AI tutorials, you would believe vector search has replaced everything else. Embed your text, store the vectors, search by similarity, done.

Reality is different. Every production RAG team eventually discovers the same thing: pure vector search misses too many things. The fix is called hybrid search, and this article explains it.

The problem with vector search

Vector search is great at finding things that mean the same. It is terrible at finding things that are the same.

Consider this real example from a customer support knowledge base:

User question: “How do I configure SAML SSO with Okta?”

Pure vector search returns:

1. “Setting up single sign-on with identity providers”
2. “Authentication best practices”
3. “Configuring user authentication in your application”

Notice what’s missing? The actual SAML/Okta-specific article. It exists in your knowledge base. Vector search ranked it lower because it talks about specific products and codes that the embedding model doesn’t fully grasp.

flowchart LR
    Q[How do I configure<br/>SAML SSO with Okta?] --> VEC[Vector search]
    VEC --> R1[General SSO doc]
    VEC --> R2[Auth best practices]
    VEC --> R3[General auth doc]
    VEC -.misses.-> RIGHT[The actual<br/>SAML/Okta article]

What vector search struggles with

Vector search consistently misses:

These all share one trait: they are exact tokens with specific meaning. Embeddings smooth them into nearby concepts, which loses precision.

The fix: bring back keyword search

The solution is older than vector search itself. BM25 is a keyword matching algorithm from the 1990s. It scores how well a document matches a query based on:

1. How many query words appear in the document
2. How rare those words are (rare words count more)
3. Document length normalization (longer documents are not unfairly favored)

flowchart LR
    Q[SAML SSO Okta] --> BM[BM25 search]
    BM --> SCAN[Scan documents for<br/>SAML, SSO, Okta tokens]
    SCAN --> RANK[Rank by frequency<br/>and rarity]
    RANK --> HIT[Found:<br/>SAML/Okta article]

BM25 is exact in a way embeddings can never be. If the query says “SAML” and a document says “SAML”, BM25 sees the match. Embeddings might score that document lower than a doc about general “authentication” because of how they interpret similarity.

Hybrid search: use both

Hybrid search means running BOTH searches in parallel and combining the results.

flowchart TB
    Q[User question]
    Q --> VEC[Vector search]
    Q --> BM[BM25 search]
    VEC --> VR[Top 50 by<br/>semantic match]
    BM --> BR[Top 50 by<br/>keyword match]
    VR --> COMBINE[Combine + re-rank]
    BR --> COMBINE
    COMBINE --> FINAL[Top 10 results]

Each search catches what the other misses:

• Vector search finds: “I forgot my login” → “How to reset your password”
• BM25 search finds: “SAML SSO Okta” → the exact SAML/Okta article
• Together: both types of question work

How to combine results: Reciprocal Rank Fusion

You have two ranked lists. How do you merge them?

The dumb way: take top 5 from each, dedupe. Works, but throws away signal.

The right way: Reciprocal Rank Fusion (RRF). It’s two lines of code.

For each document, compute:

score = 1/(k + vector_rank) + 1/(k + bm25_rank)

Where k is a constant (usually 60), vector_rank is its rank in vector results, bm25_rank is its rank in BM25 results. Higher score = better match overall.

flowchart LR
    DOC[Document X]
    DOC -.rank 3 in vector.-> SV["1/(60+3) = 0.0159"]
    DOC -.rank 7 in BM25.-> SB["1/(60+7) = 0.0149"]
    SV --> SUM["Combined: 0.0308"]
    SB --> SUM

The beauty of RRF:

• Parameter-free (no weights to tune)
• Works with any two ranking systems
• Punishes documents that appear in only one list
• Rewards documents that appear in both

Use it. Almost nobody beats it.

Metadata filters: the secret third ingredient

The single biggest win on top of hybrid search is metadata filters.

When a user asks “How do I update my billing on the Enterprise plan?”, you don’t want to search the entire knowledge base. You want to search only:

• The billing section
• Documents tagged “Enterprise”
• Documents updated in the last 6 months

flowchart TB
    Q[Update billing on<br/>Enterprise plan]
    Q --> EXTRACT[Extract filters:<br/>section=billing<br/>plan=enterprise]
    EXTRACT --> FILTER[Filter documents<br/>before search]
    FILTER --> CANDIDATES[5,000 → 200 candidates]
    CANDIDATES --> HYBRID[Hybrid search<br/>over candidates]
    HYBRID --> RESULTS[Top results]

This is a 10x speedup AND a quality improvement. The filter narrows the search to relevant documents before similarity even matters.

How to get filters:

1. Tag your documents at ingestion (section, date, author, plan, language)
2. Extract filters from the query using a small LLM call or rules
3. Apply filters before search in your vector DB (Qdrant, Pinecone, Weaviate, pgvector all support this)

What the full pipeline looks like

flowchart TB
    Q[User question]
    Q --> EXTRACT[Extract metadata<br/>filters from query]
    EXTRACT --> PARALLEL{Parallel search}
    PARALLEL --> VEC[Vector search<br/>with filters]
    PARALLEL --> BM[BM25 search<br/>with filters]
    VEC --> RRF[RRF fusion]
    BM --> RRF
    RRF --> RERANK[Cross-encoder<br/>re-ranker]
    RERANK --> LLM[Send top 5<br/>to LLM]

This is roughly what every production RAG team converges to. The exact details vary, but the shape is always the same: filters first, hybrid search second, re-ranking third.

How to implement hybrid search

The easiest path is to use a vector database that supports it natively:

If you are not picky about hosting, Qdrant or Weaviate give you hybrid search out of the box for free. If you already use Postgres, you can combine pgvector (semantic) with tsvector (keyword), more code but no extra service.

Common mistakes

Mistake 1: Skipping hybrid search “for simplicity.” The complexity cost is small. The quality improvement is large. Always add it before you ship to production.

Mistake 2: Tuning weights between vector and BM25. RRF is parameter-free. If you find yourself fiddling with vector_weight = 0.7, bm25_weight = 0.3, stop and use RRF.

Mistake 3: Forgetting metadata filters. Filters often beat any retrieval algorithm tweak. Tag your documents well.

Mistake 4: Not stop-wording your BM25 index. “The”, “of”, “to” should be removed from the BM25 index. Most libraries do this by default; check yours.

Mistake 5: Using BM25 alone for everything. Hybrid means both. BM25 alone misses paraphrases. Vector alone misses exact matches. You need both.

Questions you will face in production

“Why not just use vector search?” Because exact-match queries (product names, error codes, acronyms, API paths) fail with pure semantic search. Try a real query like SAML/Okta against pure vector search and watch it miss the obvious answer.

“How would you combine the two ranked lists?” Reciprocal Rank Fusion. Explain the formula. Mention you’d prefer it over weighted-sum because it’s parameter-free and robust.

“What if the user types just a single word?” Short queries are where BM25 dominates over vector search. Your hybrid system handles this gracefully because BM25 contributes more signal when there are few words to embed.

“How do you handle multi-lingual queries?” Two options: (1) translate the query and search in the target language, or (2) use a multilingual embedding model. BM25 needs per-language tokenization either way.

“How much does hybrid search slow you down?” Done in parallel, almost not at all. Both searches run simultaneously, then a fast fusion step combines them. You typically add 10-30ms.

What to remember

• Pure vector search misses exact-token matches (product names, codes, acronyms)
• BM25 handles those
• Combine them in parallel; merge with Reciprocal Rank Fusion
• Add metadata filters first, they often beat any algorithm change
• Use a vector DB that supports hybrid search natively (Qdrant, Weaviate)
• Hybrid search is the production default. Always add it.

What to study next

1. The original BM25 paper (Robertson and Zaragoza, 2009), short, foundational
2. Pinecone’s “Hybrid Search 101” article, practical walkthrough
3. Anthropic’s “Contextual Retrieval” blog post, shows hybrid + contextual on real benchmarks

Build a tiny RAG. Try pure vector. Try hybrid. The improvement is visible in 30 queries.

Next in the curriculum: article 07, Embeddings Deep Dive: what they are, why they work, and how to benchmark them.

Further reading

• Robertson & Zaragoza (2009) — The Probabilistic Relevance Framework: BM25 and Beyond. The canonical BM25 reference, 1-line formula explained.
• Cormack, Clarke, Büttcher (2009) — Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods. The RRF paper. Two pages, surprisingly readable.
• Anthropic — Contextual retrieval (uses hybrid). Production team showing exactly the hybrid pipeline described here, with benchmark numbers.
• Elasticsearch — Practical BM25. Friendliest explanation of BM25 internals on the web.

Where this article comes from. This is a synthesis of common practice in AI engineering as of 2026, not a citation of any single paper. The sources above are where the broad mechanics and specific numbers come from. If you find an error or have a better source for a claim, the article gets fixed within a day, send me a note.