Embeddings Deep Dive

In the RAG article we said embeddings turn text into numbers. That is true. It is also the level of explanation that lets you build something that works but does not let you debug it when it does not.

This article goes one level deeper. By the end you will know what embeddings are mathematically, how to pick a model, how to benchmark on your own data, and when fine-tuning is worth the effort.

What an embedding really is

An embedding is a vector. A list of numbers, usually 384 to 3,072 long, that represents the meaning of a piece of text.

flowchart LR
    T1["'I love this product'"] --> M[Embedding Model]
    M --> V1["[0.12, -0.45, 0.78, ...]<br/>length 1024"]

Three properties matter:

1. Fixed length. A 10-word document and a 10,000-word document both become a vector of the same size. The model compresses the meaning into a fixed shape.
2. Distance has meaning. Similar text → vectors close together. Different text → vectors far apart. “Close” is usually measured by cosine similarity.

3. The model decides the meaning. Two embedding models produce different vectors for the same text. Vectors are not portable between models.

That third one trips up beginners. If you embed your knowledge base with model A and you embed queries with model B, you get garbage. Always use the same model for both indexing and querying.

How models are trained

Knowing roughly how embedding models are trained helps you reason about their behavior.

Most modern embedding models use contrastive learning:

flowchart TB
    PAIRS[Training data: text pairs] --> POS[Positive pairs:<br/>question + correct answer]
    PAIRS --> NEG[Negative pairs:<br/>question + unrelated text]
    POS --> TRAIN[Train model to:]
    NEG --> TRAIN
    TRAIN --> G1[Pull positive vectors closer]
    TRAIN --> G2[Push negative vectors apart]

The model sees millions of pairs. After training, it has learned that “question and its correct answer” should have similar vectors, while “question and random text” should not.

Why this matters:

• Models trained on questions answer questions well. This is the default for retrieval.
• Models trained on code understand code. Pick a code-specific model if you embed code.
• Models trained on one language know that language best. Multilingual models exist; they trade off some accuracy for breadth.

The models you actually use in 2026

Choosing between them: benchmarks first, cost second, hosting third.

Benchmarking on YOUR data

This is the single most undervalued skill in production RAG. MTEB and other public benchmarks are useful for ranking models broadly, but they overfit. The model that wins on MTEB might lose on your data.

The setup takes one afternoon:

flowchart LR
    DATA[50 to 200<br/>real queries from prod]
    DATA --> GT[Label: which doc<br/>is the right answer<br/>for each query]
    GT --> EVAL[Eval loop:<br/>for each model,<br/>compute recall@10]
    EVAL --> WIN[Model with<br/>highest recall@10]

Step 1: Gather queries. Pull 50-200 real questions users ask (or you predict they will).

Step 2: Label answers. For each query, mark which documents in your corpus contain the answer. This is the only manual step. Plan one hour per 50 queries.

Step 3: For each candidate model:

• Index your full corpus with that model
• Embed each query
• Retrieve top 10 by cosine similarity
• Compute recall@10: how often the labeled correct doc is in the top 10

Step 4: Compare. The model with highest recall@10 wins. Usually the difference between candidate models is 5 to 20 percentage points, which is huge.

Code skeleton:

import numpy as np

def recall_at_k(query_embeddings, doc_embeddings, ground_truth, k=10):
    correct = 0
    for q_emb, true_doc_ids in zip(query_embeddings, ground_truth):
        # cosine similarity
        scores = doc_embeddings @ q_emb / (
            np.linalg.norm(doc_embeddings, axis=1) * np.linalg.norm(q_emb)
        )
        top_k = scores.argsort()[-k:]
        if any(doc_id in true_doc_ids for doc_id in top_k):
            correct += 1
    return correct / len(query_embeddings)

Run this for each candidate. Pick the winner. Repeat every 6 months because new models keep coming out.

Dimensionality and storage

Bigger vectors are not always better. They cost more to store and slower to search.

OpenAI’s text-embedding-3 models support Matryoshka embeddings: you can truncate the vector to a smaller dimension and lose only a few percent of quality. If 3072 dimensions is overkill for your corpus size, truncate to 1024 or 512.

Storage formats:

• float32: 4 bytes per dimension. Default.
• float16: 2 bytes per dimension. Half the storage. Negligible quality loss.
• int8 (quantized): 1 byte per dimension. 4x less storage. Slight quality loss. Worth it at scale.
• Binary (1 bit per dim): 32x less storage. Big quality loss. Only for very large corpora.

For a 10M-document corpus on text-embedding-3-large quantized to int8, you go from 120 GB to 30 GB. That changes which databases are practical.

When to fine-tune

The default answer for “should I fine-tune my embeddings?” is no.

Fine-tuning is worth it when:

• Your domain language is genuinely unusual (medical, legal, very specific technical jargon)
• You have at least 1,000 labeled query-document pairs (or can generate them)
• You have benchmarked off-the-shelf models and the best one is still under your accuracy bar

Fine-tuning is NOT worth it when:

• You just want better retrieval; try a bigger off-the-shelf model first
• You have fewer than a few hundred labeled pairs
• The off-the-shelf model is “close enough”

The right move: spend time on hybrid search, re-ranking, and chunking strategy before fine-tuning embeddings. The leverage is higher.

flowchart TB
    START[Retrieval quality<br/>not good enough]
    START --> Q1{Tried hybrid<br/>search?}
    Q1 -->|No| HY[Add BM25 + RRF]
    HY --> END[Re-evaluate]
    Q1 -->|Yes| Q2{Tried a<br/>re-ranker?}
    Q2 -->|No| RR[Add cross-encoder<br/>re-ranker]
    RR --> END
    Q2 -->|Yes| Q3{Tried better<br/>chunking?}
    Q3 -->|No| CH[Structural or<br/>semantic chunking]
    CH --> END
    Q3 -->|Yes| FT[Now consider<br/>fine-tuning]

Contextual retrieval, the 2024 trick

Anthropic published a clever technique that often beats fine-tuning for far less effort.

The idea: before embedding each chunk, prepend a one-sentence context that explains where the chunk comes from.

Original chunk: "Click Settings > Security > Reset Password."

Contextualized chunk: 
"This is from the user account management guide. Section on password recovery.
Click Settings > Security > Reset Password."

The added context makes embeddings far more searchable. Anthropic reports 35-50% improvement in retrieval accuracy.

How to generate the context: use an LLM with the full document as context, asking for a one-sentence preface per chunk. It costs more at index time but is permanent (you embed once).

If you are going to invest engineering time in retrieval, this is where to invest it first.

Common beginner mistakes

Different models for indexing and querying. You will get garbage. Same model for both, always.

Embedding the whole document. Embed chunks (400-800 tokens). The model averages too much meaning into a single vector if the document is long.

Trusting MTEB blindly. Benchmark on your own data. Differences in domain are larger than differences in benchmark scores.

Forgetting to update embeddings after model changes. If you switch embedding models, you re-embed the entire corpus. This is not a “rolling update.”

Treating cosine similarity as a quality score. A high similarity does not mean “this is a good answer.” It means “this is similar text.” Use re-ranking on top.

Storing embeddings in your main Postgres without pgvector. It will work for 10k vectors. It will fall over at 1M. Use a vector-aware index (pgvector with HNSW or IVF, or a dedicated vector DB).

Questions you will face in production

“Which embedding model should I use?” Whichever wins on your benchmark of your data. Default to a 1024-dim model from OpenAI/Voyage/Cohere if you have no benchmark yet.

“How often do we re-embed the corpus?” Whenever you change embedding models. Otherwise: every time a source document changes. Treat embeddings as derived data, like a search index.

“Should we host embeddings ourselves or use an API?” API until your monthly embedding bill exceeds your engineer’s time-cost of hosting. Roughly: API at <$500/mo, consider hosting at >$2k/mo.

“How do we handle PII in the embedded text?” Embeddings are not reversible to plaintext directly, but a determined attacker with the model can recover information. Treat them like the original text. Encrypt at rest. Apply the same access controls.

“What is the maximum input size?” Each model has a max input. OpenAI’s text-embedding-3 is 8,191 tokens. Voyage’s is 32k. If your chunks exceed it, you have to re-chunk.

What to remember

• Embeddings are fixed-length vectors. Distance has meaning.
• Same model for index and query. Always.
• Benchmark on YOUR data. MTEB is a starting point, not the answer.
• Cosine similarity is the standard metric.
• Dimensionality and quantization are real cost levers.
• Fine-tune as a last resort. Try hybrid search, re-ranking, chunking first.
• Contextual retrieval (prepending one sentence per chunk) often beats fine-tuning.

What to study next

You now have a strong foundation in retrieval. The next gap is evaluating whether your retrieval (or the whole pipeline) is actually working.

Move to article 08, LLM Evaluation Pipelines, or article 09, LLM-as-Judge.

Further reading

• Muennighoff et al. (2022) — MTEB: Massive Text Embedding Benchmark. The benchmark most embedding models report scores on. Useful, but overfit; always benchmark on your own data.
• OpenAI — text-embedding-3 announcement (Matryoshka). How Matryoshka representation works in practice.
• Kusupati et al. (2022) — Matryoshka Representation Learning. The paper behind Matryoshka embeddings.
• Reimers — Sentence Transformers documentation. Original library that popularized embedding-based retrieval; great explainers on the math.

Where this article comes from. This is a synthesis of common practice in AI engineering as of 2026, not a citation of any single paper. The sources above are where the broad mechanics and specific numbers come from. If you find an error or have a better source for a claim, the article gets fixed within a day, send me a note.