One vector per 32K-token document finds a detail 30% of the time; chunks find it 56% of the time
Chunking means splitting a document into pieces. UltraBERT, with a 32,000-token window, promised to make it unnecessary by fitting the whole text into one vector. Qdrant tested this on essays with a single off-topic sentence inserted: at 32K tokens, the whole-document vector found the right essay 30% of the time, while 512-token chunks found it 56% of the time. At 256 tokens, the same vector managed 84%. For finding details, chunks remain the foundation.
AI-processed from Qdrant Blog; edited by Hamidun News
Anyone who has built document search or RAG has run into the same question: how to cut the text into pieces. If a piece is too small, it loses context. If it is too large, the embedding model truncates it. UltraBERT from DMIS Lab promised to solve this: it accepts up to 32,000 tokens and produces a single 768-dimensional vector for an entire document. In that world, chunking seems unnecessary. The Qdrant team tested this in practice. The short answer: it is still needed.
What exactly was compared
UltraBERT is a dense embedding model with 151 million parameters. It is built on an English encoder trained for 32K-token inputs, so most documents pass through in a single pass.
Qdrant compared four setups: one vector per whole document, 512-token chunks, both approaches combined, and all of the above in compressed form. The test documents are long, averaging between 7,600 and 13,800 tokens. The tasks varied: finding a TV episode transcript from its plot summary, answering a question that requires several Wikipedia articles, and locating the right fragment of a meeting. There were also two synthetic tests: one hid a password in a text, the other inserted one off-topic sentence into an essay and asked for the essay based on a question about that sentence. For large-scale checks, the team used 200,000 English Wikipedia articles and 800 questions, each about one article.
Chunks still find the details
The whole-document vector does read all 32K tokens. In the password task, it ranked the correct document first in 99% of cases across all lengths, and at 32K tokens it never made a mistake. But in most of the other tasks, chunks were more accurate.
Chunks won where the answer is spread across several paragraphs: on 2WikiMultihopQA, QMSum, and MLDR. The whole-document vector won only on SummScreenFD, where the query describes an entire episode.
The off-topic sentence test shows why. Imagine a long essay with one unrelated phrase inserted. A single vector cannot retain every such detail. At 256 tokens, the whole-document vector found the right essay in 84% of cases; at 32K tokens, only in 30%. Chunks of 512 tokens at the same length achieved 56%.
The hybrid, which searches both the whole-document vector and the chunks and merges the results, helped only slightly. On SummScreenFD and Needle, the gain was up to 0.02 and 0.03 on the metric scale, that is, 2–3 points out of 100. On 2WikiMultihopQA, QMSum, and MLDR, the hybrid performed worse than chunks alone, most notably on MLDR.
What chunks cost and how to compress them
The main argument against chunks is the number of vectors. MLDR articles split into an average of 7 chunks, producing 1.43 million vectors in total. In float32, that is about 4 GB.
Quantization compresses each vector from 3,072 bytes to 104, a 30x reduction. All 200,000 articles take about 140 MB of RAM. Each chunk was stored as a separate point in Qdrant, and a grouped query returned each article only once.
Compared with exact search, the compressed vectors retained 95% of the quality, and 98% with rescoring against the original vectors. Uncompressed vectors in the same index achieved 99%. Rescoring has a cost: in the test, the original vectors were kept on disk, and reading them added 4–11 ms per query. While the disk cache was cold, latency exceeded 100 ms.
The whole-document vector compresses worse than chunks. On MLDR, at 1 bit it kept 89% of the quality. At 2 bits, it matched the uncompressed vector in the same index. So it is best to start with 2 bits or keep rescoring.
What to do in practice
Whole documents are also more expensive to process. Embedding the full SummScreenFD transcripts took roughly 3 times as long as embedding 512-token chunks on an NVIDIA L4.
Qdrant's recommendations:
- For detail retrieval, start with 512-token chunks.
- For questions about single facts, try 256.
- Compression makes extra vectors more affordable, but check quality and rescoring latency on your own queries.
UltraBERT has another use. You can search whole documents first to identify the topic, then search for the answer within the chunks of the documents found. The authors consider this two-stage scheme worth testing, especially since simply merging the two searches consistently did not help.
Limitations of the study: the tests were in English and on specific benchmarks, and your numbers on your own corpus may differ. But the takeaway is clear: a 32K-token window gives more freedom in choosing what one vector represents, but it does not eliminate the need to split text.
- May 21, 2026RAG Systems Break on Real Data: Retrieval Is to Blame, Not the Model
- Jan 25, 2024OpenAI Cuts Embedding Costs 5x for RAG: $0.02 Instead of $0.10 per Million Tokens
- Apr 30, 202610 RAG Approaches That Actually Work in Production: From Basic to GraphRAG
- Oct 9, 2026RAG Over 11,000 Files Confused Part Numbers and Returned Data From the Wrong Machine
Key Questions Answered
What exactly was compared?
UltraBERT is a dense embedding model with 151 million parameters. It is built on an English encoder trained for 32K-token inputs, so most documents pass through in a single pass. Qdrant compared four setups: one vector per whole document, 512-token chunks, both approaches combined, and all of the above in compressed form. The test documents are long, averaging between 7,600 and 13,800 tokens.
What are the key details on Chunks still find the details?
The whole-document vector does read all 32K tokens. In the password task, it ranked the correct document first in 99% of cases across all lengths, and at 32K tokens it never made a mistake. But in most of the other tasks, chunks were more accurate. Chunks won where the answer is spread across several paragraphs: on 2WikiMultihopQA, QMSum, and MLDR. The whole-document vector won only on SummScreenFD, where the query describes an entire episode. The off-topic sentence test shows why.
What chunks cost and how to compress them?
The main argument against chunks is the number of vectors. MLDR articles split into an average of 7 chunks, producing 1.43 million vectors in total. In float32, that is about 4 GB. Quantization compresses each vector from 3,072 bytes to 104, a 30x reduction. All 200,000 articles take about 140 MB of RAM. Each chunk was stored as a separate point in Qdrant, and a grouped query returned each article only once.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.