RAG Knowledge Base: 100 GB of Files Yield 1.5 Million Chunks and 9 Hours of Loading
Imagine a company with 100 GB of policies, contracts and instructions that it needs to load into an AI-powered corporate knowledge base (a RAG setup: retrieval of document passages plus a language model). The files turn into roughly 1.5 million text chunks, and processing them on a single GPU takes about 9 hours. At 10 TB, that becomes 38 days. This is why hardware is sized by chunk count rather than gigabytes, and why the work starts with data cleanup rather than model selection.
AI-processed from Habr AI; edited by Hamidun News
A company wants employees to ask questions about internal documents in plain language and get an answer with a link to the source. The usual solution is a knowledge base built on RAG: the system finds the relevant pieces of documents and feeds them to a language model, which writes the answer. The architect's main question is how much this will cost in hardware.
The answer is unexpected: the size of the environment is determined not by the model but by the number of chunks the documents are cut into. In the guide authors' example, 100 GB of files become 1.5 million chunks (pieces of text of about 500 tokens each), and loading them on a single GPU takes about 9 hours.
For 10 TB, that is already 150 million chunks and 38 days. The guide comes from the ITGLOBAL.COM team.
It is vendor material, and the measurements were made on the vendor's own infrastructure, but as a checklist it is useful to anyone building such a system.
Seven links in the chain and where they break
The language model does not know your documents, and fine-tuning it after every policy edit is expensive. So the model is not trained on them: for each question, it is shown the retrieved chunks. Update a document, and the answer updates with it. Say an employee asks "how to file for unpaid leave," while the policy says "leave without retention of salary." Keyword search will miss; semantic search will find the right paragraph.
The pipeline has seven links, and answer quality is set by the weakest one. Most often that is retrieval, not the model.
- Document parsing (Docling, Apache Tika, Unstructured, OCR for scans). Tables fall apart, Russian scans get read as Latin characters, and headers and footers end up in every chunk.
- Chunking. Documents are split by headings and paragraphs with a small overlap, and the document title is appended to each chunk. Without it, a "compensation table" from two different SLAs looks identical.
- Embedder (Qwen3-Embedding, BGE-M3, multilingual-e5, GigaEmbeddings, Gemini Embedding). It is often chosen from a global leaderboard and never tested on the company's own Russian documents.
- Vector database (pgvector, Qdrant, Milvus, Weaviate, OpenSearch). Typical mistakes: no access-rights filter, and no calculation of the memory the index needs.
- Keyword search (BM25, PostgreSQL full-text search). It is needed for contract numbers and error codes. Keyword and semantic scores cannot be added together without normalizing them to a single scale; otherwise one search drowns out the other.
- Reranker (Qwen3-Reranker, bge-reranker, Voyage rerank, Cohere Rerank). It takes the 20–50 retrieved chunks and orders them more precisely. Call it without the right query template and the ranking becomes nearly random.
- Answer language model (Qwen, GigaChat, YandexGPT, foreign models). It must be explicitly forbidden from answering "off the top of its head" when no relevant chunk exists.
On top of this, you need access rights and a set of test questions with known answers. Only four components require a GPU: scan parsing, the embedder, the reranker and the language model. Everything else runs on ordinary servers.
Which vector database to choose
- pgvector — if you already have PostgreSQL and up to tens of millions of chunks. Access rights are checked with plain SQL, but the index supports vectors of up to 2000 dimensions, and the access-rights filter is applied after the search.
- Qdrant — a standalone database for millions to hundreds of millions of chunks. The access-rights filter works during the search itself, and vectors are compressed 4–32 times.
- Milvus — hundreds of millions to billions of chunks, but it needs an operations team: Kubernetes, etcd, object storage.
- Weaviate — when you need hybrid search out of the box. Compression is off by default, and Russian morphology is not declared as supported.
- OpenSearch — if you already run full-text search on it. It is heavy for vector search alone.
The authors' rule of thumb: up to 10 million chunks, all five will cope; from 10 to 200 million, Qdrant or OpenSearch with compression; beyond that, Milvus or a Qdrant cluster.
Two traps. First: pgvector stores the 2560-dimension vector of Qwen3-Embedding-4B but does not index it. The workarounds are half precision or a truncated vector, for example 1024 dimensions. Second: if an employee can access only a small part of the base and the access-rights filter is applied after the search, they will get an empty answer. The permissions scheme should be designed together with the choice of database.
Sizing the environment and checking the data
In the authors' example, text makes up 5% of the file volume, duplicates account for 30%, and a chunk is 500 tokens with a 15% overlap. The result: 100 GB of files yield 1.5 million chunks, 23 GB of memory for vectors of 2560 dimensions, and about 9 hours of loading at an embedder speed of 20–22 thousand tokens per second.
1 TB means 15 million chunks and 4 days; 10 TB means 150 million and 38 days. Compressing vectors to int8 and truncating them to 1024 dimensions cuts memory 10 times: for 10 TB, that is 230 GB instead of 2.3 TB.
These coefficients are illustrative; the real shares of text and duplicates should be measured on a sample of 1–2% of the files.
Database size barely matters for answering queries. The question vector is computed in 0.19 s, reranking 20 chunks takes 0.68 s, and most of the time goes to the language model. For models with 26–27 billion parameters, a single 80–96 GB card is enough, while eight H200s for GLM-5 are rarely justified for a knowledge base.
The main recommendation is to start with the data. Gartner says 63% of organizations are not confident in their data management practices. Before writing any code, you need to list the sources with their owners, estimate duplication (by practitioners' estimates, about 30% outdated versions and 20% duplicates), check scan quality (below 300 dpi, expect errors), settle access rights and 152-FZ questions, and collect at least 30, preferably 100–200, real questions with reference answers. Connect half as many files, and everything in the estimate halves.
- Nov 19, 2026AI Models Added a School to the Target List in 70% of Vals AI Tests
- Nov 19, 2026Q.ANT and AMD Run a 1B-Parameter LLM With an Optical Chip in Place of Attention
- Nov 19, 2026The Word "Video Game" Raised GPT-6 Astra's Willingness to Pick Targets From 6% to 92%
- Nov 19, 2026AWS adds a leaderboard of 70+ models to SageMaker: quality, speed and price in one table
Key Questions Answered
What are the key details on Seven links in the chain and where they break?
The language model does not know your documents, and fine-tuning it after every policy edit is expensive. So the model is not trained on them: for each question, it is shown the retrieved chunks. Update a document, and the answer updates with it. Say an employee asks "how to file for unpaid leave," while the policy says "leave without retention of salary." Keyword search will miss; semantic search will find the right paragraph. The pipeline has seven links, and answer quality is set by the weakest one.
Which vector database to choose?
pgvector — if you already have PostgreSQL and up to tens of millions of chunks. Access rights are checked with plain SQL, but the index supports vectors of up to 2000 dimensions, and the access-rights filter is applied after the search. Qdrant — a standalone database for millions to hundreds of millions of chunks. The access-rights filter works during the search itself, and vectors are compressed 4–32 times.
What are the key details on Sizing the environment and checking the data?
In the authors' example, text makes up 5% of the file volume, duplicates account for 30%, and a chunk is 500 tokens with a 15% overlap. The result: 100 GB of files yield 1.5 million chunks, 23 GB of memory for vectors of 2560 dimensions, and about 9 hours of loading at an embedder speed of 20–22 thousand tokens per second. 1 TB means 15 million chunks and 4 days; 10 TB means 150 million and 38 days.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.