243,000 texts for a 29M-parameter model: how the author picked data from Wikipedia and psychology books

The author of a Habr series, who does not consider himself a programmer, is preparing data for a small 29M-parameter model. The number in the title, 243,000, is how many texts remain in the training set. The author takes 200,000 paragraphs of Russian Wikipedia and 50,000 paragraphs from psychology books. After removing duplicates he has 248,379, of which 4,967 go to quality validation and 243,412 stay for training.

Why these texts are needed. An embedding is a vector of numbers into which a text is turned so a machine can compare texts by meaning. The small model (the student) learns to output the same vectors as the large one (the teacher). This technique is called distillation. The goal of the ten-part series is isotropic embeddings, that is, vectors that fill space evenly instead of clumping into a narrow bundle. In part two the author works through the data: where to get it, how to clean it and how much is actually needed. All the code was written by the DeepSeek neural network, while the author handled debugging and fixing errors.

Wikipedia from a 380 MB archive: parsing in a minute

Two kinds of text are needed. The general corpus gives the student broad topic coverage: without it the model would overfit to a narrow area and perform poorly on unfamiliar texts. The domain corpus on psychology teaches it the specifics of the task the whole project was started for.

The full Russian Wikipedia dump weighs about 5.5 GB compressed, which is a lot for Colab with its limited disk. The multistream version helps: the same dump, cut into hundreds of small files. The author downloaded one of them, ruwiki-latest-pages-articles-multistream1.xml-p1p224167.bz2, at 380 MB. For 250,000 texts, 100–200 MB of clean text would have been enough, but the result was 1.24M paragraphs and 835.9 MB.

The text was extracted with the WikiExtractor utility, using the --no-templates flag to avoid pulling in wiki templates, and four processes. Speed: 1,185 articles per second, about 76,000 articles in 64 seconds.

Three kinds of junk remain afterward:

• section headings like == История ==;
• empty parentheses () where templates were cut out;
• the first line inside <doc>, which duplicates the article title.

The script cleans all of this with regular expressions and discards paragraphs shorter than 40 characters. A side effect: instead of "Площадь — 65 300 км²" what remains is "Площадь — км²". The author considers this harmless: numbers get split into subtokens (small word fragments) anyway, and sentence context matters more for an embedding.

Psychology: 587 files and a ten-word threshold

The domain corpus is built from a personal collection of books in .md format: psychology, psychotherapy, psychiatry, mathematical statistics for psychologists, anatomy and physiology. The author does not publish the texts themselves. The script strips Markdown markup and HTML tags, joins words split by hyphenation and keeps only lines of at least ten words. This removes page numbers, running headers and stray headings.

Result:

• 587 files;
• 382,327 paragraphs;
• about 18.9M words;
• 258.4 MB.

Why 243,000 of 1.36M texts are used

First comes a filter on length (60–800 characters) and share of letters (at least 55%), which throws out table and formula fragments. After it, Wikipedia has 1,056,981 paragraphs and psychology 330,714. The author keeps the full corpus of 1,359,942 unique texts in reserve: if the data falls short, he will not have to parse Wikipedia again.

The training set is deliberately trimmed: 200,000 Wikipedia and 50,000 psychology paragraphs with a fixed seed (the initial value of the random number generator). After deduplication, that is, removal of repeats, 248,379 texts remain, 49,484 of them psychological. If a line appears in both sources, it is assigned to psychology. Then 2% (4,967 texts) goes to validation, that is, quality checking, and 243,412 stay in training. The indices of domain rows in the training part are stored in a separate array, domain_indices.npy: there are 48,477 of them, and the rest ended up in validation. This lets the author evaluate quality on psychology separately later.

The reason for the cut is cost. A three-layer student trains on a T4 GPU in Colab. One epoch (a full pass over the data) with batch_size=64 on the full set takes 15–20 minutes, and at 3–10 epochs that adds up to hours. The biggest expense is the teacher: it must compute embeddings for every training text, and even on a GPU with 1.33M examples that takes tens of minutes. A 243,000-text sample is 5–6 times cheaper and enough for a first cycle.

What an engineer can take from this

• Small parts of the Wikipedia multistream dump save you from downloading 5.5 GB and suit a limited disk.
• Keeping the full corpus on disk is cheap: resampling it with different quotas takes 28 seconds.
• A domain share of about 20% in training is a reasonable benchmark for distillation on a narrow task.
• The approach has flaws. The first script holds about 800 MB in memory, and the size of corpus.txt is given in the text with two different figures (753.8 and 718.9 MB). For a Colab experiment that is tolerable.

Next in the series: the teacher's embeddings and training the student.