Chunking

Chunking is the process of splitting source documents into smaller text segments before embedding and storing them in a vector database, enabling efficient and precise retrieval in RAG and search systems.

Chunking is the process of dividing large source documents into smaller, self-contained text segments—called chunks—before converting them into vector embeddings and storing them in a vector database. Because embedding models produce a single fixed-size vector per input and language model context windows have token limits, embedding an entire long document collapses fine-grained meaning into a single representation, making precise retrieval impractical without prior segmentation.

The choice of chunking strategy significantly affects retrieval quality. Fixed-size chunking splits text at a set token count, typically 256–1024 tokens, often with a sliding overlap of 10–20% to preserve context across segment boundaries. Sentence- and paragraph-based chunking respects natural linguistic units. Semantic chunking identifies topic-shift boundaries within text and splits there rather than at arbitrary character counts. Hierarchical chunking stores both fine-grained and coarse-grained representations of the same content, enabling retrieval at multiple levels of granularity. Frameworks such as LangChain and LlamaIndex expose these strategies as configurable options with adjustable parameters.

Chunk size creates a direct tradeoff: smaller chunks yield higher retrieval precision—returned text is tightly relevant—but may omit necessary surrounding context; larger chunks provide more context but dilute relevance scores when only part of the chunk matches the query. This makes chunk size and overlap tunable hyperparameters that practitioners optimize empirically for each corpus and task.

As of 2026, proposition-level chunking—where each chunk represents a single factual claim extracted from the source—has shown strong performance on retrieval benchmarks. Tools for automated chunking optimization are emerging as part of RAG evaluation pipelines. The availability of models with context windows of one million or more tokens has not eliminated chunking, because focused, semantically coherent segments still outperform undifferentiated full-document retrieval for most query types.