243 thousand texts for a small 29M-parameter model: how the author selected data from Wikipedia and psychology books

The author of a Habr series, who does not consider themselves a programmer, is preparing data for a small 29M-parameter model. The number in the title, 243 thousand, is how many texts remain in the training set. The author takes 200 thousand paragraphs from Russian Wikipedia and 50 thousand paragraphs from psychology books. After deduplication, 248 379 remain; 4 967 of them go to quality validation and 243 412 stay for training.

Why these texts are needed. An embedding is a vector of numbers into which a text is turned so that a machine can compare texts by meaning. The small model (the student) learns to produce the same vectors as the large one (the teacher). This technique is called distillation. The goal of the ten-part series is isotropic embeddings: vectors that fill the space evenly instead of clumping into a narrow bundle. In part two, the author works through the data: where to get it, how to clean it, and how much is actually needed. All the code was written by the DeepSeek neural network, while the author handled debugging and fixing errors.

Wikipedia from a 380 MB archive: parsing in a minute

Two kinds of text are needed. The general corpus gives the student broad topic coverage; without it, the model would overfit to a narrow domain and perform poorly on unfamiliar texts. The domain corpus on psychology teaches the model the specifics of the task the whole project is built around.

The full Russian Wikipedia dump weighs about 5.5 GB compressed, which is a lot for Colab with its limited disk. The multistream version helps: it is the same dump, but split into hundreds of small files. The author downloaded one of them, ruwiki-latest-pages-articles-multistream1.xml-p1p224167.bz2, at 380 MB. For 250 thousand texts, 100–200 MB of clean text would have been enough, but the result was 1.24 million paragraphs and 835.9 MB.

The text was extracted with the WikiExtractor utility, using the --no-templates flag to avoid pulling in wiki templates, and four processes. Speed: 1185 articles per second, about 76 thousand articles in 64 seconds.

Three kinds of junk remain after extraction:

• section headings like == История ==;
• empty parentheses () where templates were cut out;
• the first line inside <doc>, which duplicates the article title.

The script cleans all of this with regular expressions and discards paragraphs shorter than 40 characters. A side effect: instead of "Площадь — 65 300 км²" ("Area — 65 300 km²"), what remains is "Площадь — км²" ("Area — km²"). The author considers this harmless: numbers get split into subtokens (small word fragments) anyway, and for an embedding the context of the sentence matters more.

Psychology: 587 files and a ten-word threshold

The domain corpus is built from a personal collection of books in .md format: psychology, psychotherapy, psychiatry, mathematical statistics for psychologists, anatomy and physiology. The author does not publish the texts themselves. The script strips Markdown markup and HTML tags, joins words split by hanging hyphens, and keeps only lines of at least ten words. This removes page numbers, running headers and stray headings.

Result:

• 587 files;
• 382 327 paragraphs;
• about 18.9 million words;
• 258.4 MB.

Why 243 thousand are taken from 1.36 million texts

First comes a filter on length (60–800 characters) and share of letters (at least 55%), which throws out fragments of tables and formulas. After it, Wikipedia has 1 056 981 paragraphs left and psychology has 330 714. The author saves the full corpus of 1 359 942 unique texts as a reserve, so that if the data runs short, Wikipedia will not need to be reprocessed.

The training set is deliberately trimmed: 200 thousand Wikipedia and 50 thousand psychology paragraphs, with a fixed seed (the initial value of the random number generator). After deduplication, i.e. removal of repeats, 248 379 texts remain, 49 484 of them psychological. If a line appears in both sources, it is assigned to psychology. Then 2% (4 967 texts) goes to validation, i.e. quality checking, and 243 412 stay in training. The indices of domain rows in the training part are stored in a separate array, domain_indices.npy: there are 48 477 of them, and the rest ended up in validation. This makes it possible to evaluate quality on psychology separately later.

The reason for the cut is cost. The three-layer student trains on a T4 GPU in Colab. One epoch (a full pass over the data) with batch_size=64 on the full corpus takes 15–20 minutes, and at 3–10 epochs that adds up to hours. The biggest expense is the teacher: it has to compute embeddings for every training text, and even on a GPU, with 1.33 million examples, that takes tens of minutes. A sample of 243 thousand is 5–6 times cheaper and sufficient for a first cycle.

What an engineer can take from this

• Small parts of the Wikipedia multistream dump spare you a 5.5 GB download and suit a limited disk.
• Keeping the full corpus on disk is cheap: it can be resampled with different quotas in 28 seconds.
• A domain share of about 20% in training is a reasonable benchmark for distillation aimed at a narrow task.
• The approach has flaws. The first script holds about 800 MB in memory, and the article gives the size of corpus.txt as two different numbers (753.8 and 718.9 MB). For a Colab experiment, this is tolerable.

Next in the series: the teacher's embeddings and training the student.