Choosing Chunking Strategies: A Practical Framework

In the RAG article we said chunking is the single most important decision in a RAG system. Get it wrong and nothing else helps. Get it right and even a mediocre embedding model gives good results.

This article explains what chunking is, why it matters so much, and the four main strategies, with a clear rule for picking one.

What is chunking, again?

When you build a RAG system, your documents are too big to feed to the LLM in one piece. A 200-page manual won’t fit. Even if it did, the LLM would not find the answer.

So you split each document into smaller pieces called chunks. The chunks become the unit of retrieval.

flowchart LR
    DOC[One document<br/>50,000 words] --> SPLIT[Chunking]
    SPLIT --> C1[Chunk 1]
    SPLIT --> C2[Chunk 2]
    SPLIT --> C3[Chunk 3]
    SPLIT --> C4[Chunk 4]
    SPLIT --> CN[...]

Each chunk is then embedded and stored in your vector database. When a user asks a question, you search for chunks, not documents.

Why chunking is so important

A good chunk:

• Contains a complete idea
• Is small enough to fit several into the LLM’s context window
• Is large enough to be meaningful on its own

A bad chunk causes one of three failure modes:

flowchart LR
    subgraph fail ["Three ways chunking ruins your RAG"]
        direction LR
        F1[Chunks too small<br/>missing context]
        F2[Chunks too large<br/>too much irrelevant info]
        F3[Chunks split mid-thought<br/>broken meaning]
    end

Every chunk you index is a tradeoff between precision (small chunks, find the exact paragraph) and recall (large chunks, have full context). The strategy you pick decides where you land.

The four chunking strategies

There are dozens of variations. They all reduce to four core approaches.

Strategy 1: Fixed-size chunking

Split the document into chunks of N characters (or tokens). Done.

flowchart LR
    DOC["The mitochondria is the<br/>powerhouse of the cell.<br/>It produces ATP through..."] --> SPLIT[Split every<br/>500 characters]
    SPLIT --> C1["The mitochondria<br/>is the powerhouse<br/>of the cell. It..."]
    SPLIT --> C2["...produces ATP<br/>through oxidative<br/>phosphorylation..."]

Pros:

• Easiest to implement
• Predictable chunk sizes
• Fast

Cons:

• Often splits sentences in half
• Ignores document structure
• Cuts code blocks and tables in the middle

When to use: prototypes, simple text, when you do not have time to do better.

Strategy 2: Recursive chunking

Split the document at natural boundaries, in this order: paragraphs first. If a paragraph is too big, split by sentences. If a sentence is too big, split by words.

flowchart TB
    DOC[Document]
    DOC --> P[Paragraphs]
    P --> CHECK1{Fits in 800 tokens?}
    CHECK1 -->|Yes| KEEP[Keep as chunk]
    CHECK1 -->|No| S[Split into sentences]
    S --> CHECK2{Fits in 800 tokens?}
    CHECK2 -->|Yes| KEEP
    CHECK2 -->|No| W[Split into smaller pieces]
    W --> KEEP

Pros:

• Respects natural language boundaries
• Works well across many document types
• Library support (LangChain, LlamaIndex both ship this)

Cons:

• Still ignores headers and document structure
• Same paragraph split arbitrarily if it is long

When to use: this is the right default for most projects. Start here.

Strategy 3: Structural chunking

Use the document’s structure, headers, sections, lists, tables, as chunk boundaries.

flowchart TB
    DOC["# Main Title<br/><br/>## Section A<br/>Paragraph...<br/><br/>## Section B<br/>Paragraph..."]
    DOC --> PARSE[Parse structure]
    PARSE --> SECA[Chunk: Section A<br/>with parent title]
    PARSE --> SECB[Chunk: Section B<br/>with parent title]

The trick: include the parent header in each chunk’s text so the chunk knows what it is about.

Pros:

• Chunks are semantically coherent
• Each chunk has natural context
• Tables and code blocks stay whole

Cons:

• Requires good document parsing
• Sections vary wildly in size (table of contents = 50 words, intro = 5,000 words)

When to use: technical documentation, well-formatted PDFs, anything with clear headings.

Strategy 4: Semantic chunking

Use an embedding model to split where the topic changes. The system reads the document sentence by sentence, embeds each one, and starts a new chunk when consecutive sentences are too different.

flowchart LR
    S1[Sentence 1] --> E1[Embed]
    S2[Sentence 2] --> E2[Embed]
    S3[Sentence 3] --> E3[Embed]
    E1 -.similar.-> E2
    E2 -.different.-> E3
    E3 --> NEW[Start new chunk here]

Pros:

• Chunks follow topic boundaries, not just punctuation
• Best results when document structure is missing
• Works well for transcripts, articles, conversations

Cons:

• Slow (embedding every sentence)
• Expensive
• Often only marginally better than recursive chunking

When to use: when other strategies fail and you have budget. Not the default.

The framework: which one to pick

Here is the decision tree.

flowchart TB
    START[Your documents] --> Q1{Do they have<br/>clear headers and structure?}
    Q1 -->|Yes| STRUCTURAL[Use Structural<br/>+ parent context]
    Q1 -->|No| Q2{Is quality critical<br/>and budget large?}
    Q2 -->|Yes| SEMANTIC[Use Semantic]
    Q2 -->|No| RECURSIVE[Use Recursive<br/>this is the default]

If you do not know yet: use recursive chunking with 400-800 token chunks and 50-100 token overlap. This wins in 80% of cases.

Chunk size: the other big decision

Once you have picked a strategy, you need a chunk size. Common ranges:

| 800-1500 tokens | Long-form answers, complex synthesis, dense technical text | | 2000+ tokens | Rare. Usually means you are using RAG when you should be using long-context. |

Default starting point: 500 tokens. Run 20 test queries against it. If the LLM keeps missing context, bump to 700. If it gets distracted by noise, drop to 350. Tune from real cases, not from theory.

Overlap: should chunks share text?

Yes, a small overlap helps. When chunks share their last few sentences with the next chunk, you avoid this failure:

flowchart LR
    subgraph nooverlap ["No overlap, broken"]
        direction LR
        C1["...the model achieves 87%"] --> C2["accuracy on the benchmark..."]
    end
    subgraph overlap ["With overlap, preserved"]
        direction LR
        D1["...the model achieves 87% accuracy"] -.shared.-> D2["87% accuracy on the benchmark..."]
    end

A 50-100 token overlap is typical. More than that wastes storage with little benefit.

Exception: if you use structural chunking with parent context, you usually do not need overlap. The parent header gives every chunk the context it needs.

The “small-to-big” pattern

Modern RAG systems use a clever trick: search with small chunks, but retrieve with big chunks.

flowchart LR
    Q[User question] --> SEARCH[Search using<br/>SMALL chunks<br/>200 tokens]
    SEARCH --> FOUND[Found small chunk]
    FOUND --> EXPAND[Expand to<br/>PARENT section<br/>1500 tokens]
    EXPAND --> LLM[Send big context<br/>to LLM]

Why this works:

• Small chunks give high search precision (you match the exact sentence)
• Big parent chunks give the LLM enough context to write a good answer

This is one of the highest-impact tricks you can apply. Most production RAG systems use it.

Common chunking mistakes

Mistake 1: Using fixed-size chunking on important docs. A character-based split will cut at the worst possible moment. Use recursive chunking instead. Same code change, much better results.

Mistake 2: Splitting code, tables, or lists. These are atomic units. A function definition split in half is useless. A table cut at row 5 is useless. Always preserve these structures.

Mistake 3: Forgetting the parent title. A chunk that says “It supports up to 10,000 concurrent connections” is useless without knowing what it refers to. Prefix every chunk with its parent header.

Mistake 4: One chunk size for all documents. A 5-page memo and a 200-page legal contract should not use the same chunk size. Vary by document type.

Mistake 5: Re-chunking every time you query. Chunk once, store the chunks, never re-chunk at query time. Re-chunking is expensive and slow.

Questions you will face in production

“How would you pick a chunk size?” Start at 500 tokens. Run 50 representative test queries. Measure retrieval recall@10. Adjust if a clear pattern emerges. Mention you would treat this as a hyperparameter to tune with evals, not guess.

“What if the same chunk is needed for two questions?” That is normal and fine. Chunks can be returned for many queries. Just make sure your re-ranker can score them differently per query.

“How do you handle code blocks and tables?” Keep them whole. Detect them during parsing and treat them as atomic chunks even if they exceed your token limit. Mention you would let chunk size go to 2x for code/table chunks.

“What about Anthropic’s contextual retrieval?” A 2024 technique: before embedding each chunk, ask an LLM to add a sentence of context (“This chunk is from the troubleshooting section about disk errors”). Improves retrieval by 35-50%. Costs more at index time but pays off at query time. Mentioning this signals depth.

What to remember

• Chunking is the highest-leverage decision in RAG
• Default to recursive chunking with 400-800 tokens + 50-100 token overlap
• Use structural chunking when documents have clear headers
• Use semantic chunking only when other strategies fail
• Always keep code blocks and tables whole
• Always prefix chunks with their parent header
• Use the small-to-big pattern: search small, retrieve big

What to study next

1. LangChain’s RecursiveCharacterTextSplitter source code, short, well-commented, see exactly how the recursive strategy works
2. Anthropic’s contextual retrieval blog post, the technique that boosts retrieval most for little extra cost
3. LlamaIndex’s SemanticSplitterNodeParser, see how semantic chunking is implemented

Build the smallest possible RAG with 3 strategies and 20 questions. Compare recall. You will form opinions fast.

Next in the curriculum: article 06, Hybrid Search: When Pure Vector Search Isn’t Enough.

Further reading

• LlamaIndex — Node parsers and chunking. Open-source implementations of every chunking strategy in this article.
• Anthropic — Contextual retrieval. How to prepend chunk-specific context to make chunks more searchable.
• Pinecone — Chunking strategies for LLM apps. Visual guide to the chunking trade-offs covered here.

Where this article comes from. This is a synthesis of common practice in AI engineering as of 2026, not a citation of any single paper. The sources above are where the broad mechanics and specific numbers come from. If you find an error or have a better source for a claim, the article gets fixed within a day, send me a note.