Habr Series Author Compresses a Russian Embedding Model from 180M to 29M Parameters
Embeddings turn texts into numeric vectors so you can find similar ones, clean out duplicates and group them. The 180M-parameter Russian model rubert-base is heavy for a server without a GPU, and it confuses "got a job" with "did not get a job". A Habr series author trains a small model to reproduce its vectors, ending up with 29.4M parameters and 118 MB. For search and deduplication on a CPU, a compact model can be built to fit your own task. Full text: https://example.invalid
AI-processed from Habr AI; edited by Hamidun News
Embeddings are numeric vectors into which texts are converted so they can be compared by meaning. Knowledge-base search, duplicate cleanup and grouping of support requests all rely on them. Good models exist for Russian, but they are often too heavy for an ordinary server without a graphics card, that is, for running on a CPU alone.
A series of ten articles has started on Habr: the author takes a large teacher model with 180M parameters (parameters are the trainable weights of a network; the more there are, the heavier and slower the model) and trains a small student with 29.4M parameters to reproduce its vectors. This is the introductory part: it contains no final metrics, but it clearly lays out the task, the constraints and the architecture.
What problem the project solves
The teacher is DeepPavlov/rubert-base-cased-sentence: 180M parameters, 12 layers, 768-dimensional vectors. The model is high quality but slow on a CPU. It has two known weaknesses.
The first is "blindness to negation". It distinguishes poorly between the phrases "got a job" and "did not get a job", even though their meanings are opposite. For deduplication and search this is a problem: the system will treat such texts as nearly identical.
The second is anisotropy. The vectors of all texts are squeezed into a narrow cone, so any two texts end up with high cosine similarity (a measure of how close two vectors are). It is hard to pick a threshold for "this is a duplicate": the deletion list ends up with tens of thousands of pairs instead of a handful. The base model sergeyzh/rubert-tiny-turbo, with its 312 dimensions and token averaging, compresses the space even more.
The constraints are strict: it must run on a CPU, take up to 300 MB, and run several times faster than the teacher. The task is not only to shrink the teacher but also to partly fix its shortcomings.
How the student is built
The student is based on rubert-tiny-turbo, a version of rubert-tiny2 pretrained contrastively (the model was trained to pull similar texts closer together and push dissimilar ones apart), so its semantics are better at the start. The tokenizer stays the original one, and the 84-thousand-token vocabulary is not expanded.
Two components are added on top of the encoder's ready-made hidden states:
- Multi-head attention pooling with 4 heads instead of simple averaging. The model learns on its own to weight the words in a text: important ones, such as "not", get more weight. The outputs of the heads are averaged.
- A linear projection 312 → 768. It is needed for training, so the student can be compared with the teacher in the teacher's space. After training, the projection can be kept or turned off: 312 dimensions give a smaller size and higher speed.
Whitening, that is, forced "smoothing" of the vector space, is not applied in the main model: the author expects contrastive training to bring the distribution closer to uniform on its own. For cases where the user's text differs strongly from the training data, an optional fit_domain() function is provided.
Training runs in two phases. In Phase A the encoder is frozen and only the new head is trained: it tries to make its vectors resemble the teacher's. In Phase B the top two layers are unfrozen, and two objectives are mixed in equal proportions: InfoNCE (telling the correct answer from the incorrect ones) and distillation (reproducing the teacher). The data consists of 243,412 texts from Wikipedia and a psychology corpus after deduplication, plus 99,999 triplets from deepvk/ru-HNP: a query, five matching texts and five "hard" non-matching ones.
What is useful here for practitioners
First, the goals are measurable in advance. Similarity to the teacher after Phase A should be at least 90%, and the final figure at least 85%. Retrieval accuracy on hard negative examples should be at least 97%, and turbo already starts at 96.8%. For deduplication the criterion is practical: only a handful of pairs should end up in auto-deletion, not tens of thousands. For clustering, the ratio of distances within clusters to distances between them should be clearly above one.
Second, the result is packaged as a working tool: a folder of about 120 MB (the weights alone are 117.8 MB) with a SemanticEmbedder class that offers encode, search, dedup, cluster and fit_domain. The student, turbo and the teacher are compared on the external ru-paraphrase-NMT-Leipzig set (5000 pairs) and on three practical tasks.
There are caveats too. The author states plainly that he is not a programmer: DeepSeek wrote all the code, and he did the debugging and error fixing. The texts were assembled from notes taken along the way, so they are rough in places. Quality figures will appear only in later parts, and they cannot be carried over to other domains without verification.
As a how-to guide, the series looks useful: it shows how to build a lightweight Russian-language model for CPUs without starting from scratch and without relying on a single metric.
- Apr 30, 2026Habr AI shows how to build a Linear Layer in C++ and CUDA in its 'From MNIST to Transformer' series
- May 2, 2026ChatGPT for Python Development: Architecture, CI/CD and a Practical Guide
- Oct 29, 2026A strong Russian model rates "I love cats" and "I don't love cats" as 70% similar
- Apr 30, 2026Habr AI walks through gradient descent in C++ and CUDA by training a model on MNIST
Key Questions Answered
What problem the project solves?
The teacher is DeepPavlov/rubert-base-cased-sentence: 180M parameters, 12 layers, 768-dimensional vectors. The model is high quality but slow on a CPU. It has two known weaknesses. The first is "blindness to negation". It distinguishes poorly between the phrases "got a job" and "did not get a job", even though their meanings are opposite. For deduplication and search this is a problem: the system will treat such texts as nearly identical. The second is anisotropy.
How the student is built?
The student is based on rubert-tiny-turbo, a version of rubert-tiny2 pretrained contrastively (the model was trained to pull similar texts closer together and push dissimilar ones apart), so its semantics are better at the start. The tokenizer stays the original one, and the 84-thousand-token vocabulary is not expanded. Two components are added on top of the encoder's ready-made hidden states: Multi-head attention pooling with 4 heads instead of simple averaging.
What is useful here for practitioners?
First, the goals are measurable in advance. Similarity to the teacher after Phase A should be at least 90%, and the final figure at least 85%. Retrieval accuracy on hard negative examples should be at least 97%, and turbo already starts at 96.8%. For deduplication the criterion is practical: only a handful of pairs should end up in auto-deletion, not tens of thousands. For clustering, the ratio of distances within clusters to distances between them should be clearly above one.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.