What Is a Chunking Strategy?

What Is a Chunking Strategy?

A chunking strategy is the rule that splits source documents into passages before they are embedded and indexed in a RAG system. The simplest strategy is fixed-size chunking — every chunk is N tokens with M tokens of overlap. More sophisticated strategies include recursive chunking (split on paragraph, then sentence, then token), semantic chunking (split at topic boundaries detected by embedding similarity), sentence-window retrieval (index small chunks but return a window around them), and parent-document retrieval (small searchable chunks point to larger parent passages for generation). The strategy sets retrieval recall and precision ceilings for the entire system; you cannot prompt-engineer your way out of bad chunking.

Why It Matters in Production LLM and Agent Systems

Chunking is the most under-instrumented decision in RAG. Teams pick a default — 512 tokens, 50-token overlap — because a tutorial said so, then spend weeks blaming the LLM when answers go wrong. The truth is most “hallucinations” in RAG are chunking errors: the right fact was split across two chunks, neither of which made the top-k, so the model confabulated. The problem is invisible until you measure chunk-level quality.

The pain shows up across roles. ML engineers see Groundedness failures even when retrieval looks “fine” because the relevant chunk got cut mid-sentence. Retrieval engineers see recall ceilings they cannot break by tuning the embedding model — the chunker is the bottleneck. Product managers see “the bot misses obvious facts” complaints on long-form policy docs. SREs see index size balloon when overlap is set too high.

In 2026, chunking has moved past “what number of tokens” into structural strategies tied to document type. PDFs with tables need layout-aware chunking. Code corpora need symbol-aware chunking. Legal docs need section-aware chunking. Agentic RAG patterns add a layer on top: the agent re-chunks dynamically when initial retrieval fails. The strategy is no longer a single config — it is a per-document-type policy that has to be evaluated and tuned independently.

How FutureAGI Handles Chunking Strategy

FutureAGI’s approach is to provide the chunk-level evaluators teams need to compare strategies empirically rather than by intuition. fi.evals.ChunkAttribution returns pass/fail on whether the generated answer references the retrieved chunks at all — a low ChunkAttribution rate means the chunks are noise that the model ignored. fi.evals.ChunkUtilization returns a 0–1 score on how effectively the model used the retrieved chunks — a low utilization score means the chunks contained relevant information but the model under-used it, often a chunk-size or chunking-boundary issue.

These pair with fi.evals.ContextRelevance (was the chunk relevant to the query?) and Groundedness (did the answer stay inside the chunks?) to give a four-signal view of chunking quality. The traceAI integrations capture chunk content as retrieval.documents attributes, so an engineer comparing two chunking strategies can A/B them on the same dataset and read off which strategy produced better evaluator scores.

A typical FutureAGI workflow: a team is choosing between recursive 512-token chunks and semantic chunking for a policy-doc corpus. They build two indexes, run the same golden dataset through each, and compare ChunkAttribution, ChunkUtilization, ContextRelevance, and RAGScore side by side. Semantic chunking shows higher ChunkUtilization but slightly lower ContextRelevance, which is exactly the trade-off semantic chunking is known for; the team picks based on which signal matters more for their use case. That is chunking-strategy decision-making with feedback in hours, not weeks of vibes-based iteration.

How to Measure or Detect It

Chunking quality is measurable per request and per index:

• fi.evals.ChunkAttribution: pass/fail on whether the answer references retrieved chunks at all — catches “model ignored the context” failures.
• fi.evals.ChunkUtilization: 0–1 score on how effectively the chunks were integrated — catches “chunks were relevant but under-used” failures.
• fi.evals.ContextRelevance: 0–1 score on whether the chunks themselves answer the query — catches retrieval-boundary errors.
• Chunk-size distribution: median, p50, p99 chunk token count — exposes outlier chunks that won’t fit in prompt.
• Top-k recall on golden queries: percentage of golden queries where the gold chunk appears in top-k — the canonical recall ceiling.
• OTel attribute: retrieval.documents content + length per request — captured by traceAI integrations.

from fi.evals import ChunkAttribution, ChunkUtilization

attribution = ChunkAttribution().evaluate(
    output="Refunds are accepted within 30 days.",
    context=["Section 4.2: Refunds may be requested within 30 days..."]
)
print(attribution.score, attribution.reason)

Common Mistakes

• Defaulting to 512-token fixed chunks for everything. Different doc types need different strategies; legal contracts and code reviews chunk differently than blog posts.
• Overlap as a free fix. Adding overlap inflates index size and recall but does nothing for boundary cuts mid-fact. Use recursive splitting at semantic boundaries instead.
• Tuning chunk size without measuring ChunkUtilization. Engineers swap 512 for 256 because a thread said so, with no metric to prove it actually helps.
• Ignoring document structure. PDFs with tables, code with symbols, legal with sections — structure-aware chunking beats generic recursive splitting on these.
• Re-chunking without re-embedding. Changing chunk size invalidates every embedding; treat chunking and embedding as one versioned artifact.