Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
InfrastructureInsight · On RadarHabr AI→ original

Habr Series Author Compresses a Russian Embedding Model from 180M to 29M Parameters

Embeddings turn texts into numeric vectors so you can find similar ones, clean out duplicates and group them. The 180M-parameter Russian model rubert-base is heavy for a server without a GPU, and it confuses "got a job" with "did not get a job". A Habr series author trains a small model to reproduce its vectors, ending up with 29.4M parameters and 118 MB. For search and deduplication on a CPU, a compact model can be built to fit your own task. Full text: https://example.invalid

AI-processed from Habr AI; edited by Hamidun News
Habr Series Author Compresses a Russian Embedding Model from 180M to 29M Parameters
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Data243,412 texts from Wikipedia and psychology sources after deduplication
Fine-tuning on 99,999 ru-HNP tripletsa query, 5 correct and 5 incorrect answers
Goalat least 85% similarity to the teacher and a model size under 300 MB

Embeddings are numeric vectors into which texts are converted so they can be compared by meaning. Knowledge-base search, duplicate cleanup and grouping of support requests all rely on them. Good models exist for Russian, but they are often too heavy for an ordinary server without a graphics card, that is, for running on a CPU alone.

A series of ten articles has started on Habr: the author takes a large teacher model with 180M parameters (parameters are the trainable weights of a network; the more there are, the heavier and slower the model) and trains a small student with 29.4M parameters to reproduce its vectors. This is the introductory part: it contains no final metrics, but it clearly lays out the task, the constraints and the architecture.

What problem the project solves

The teacher is DeepPavlov/rubert-base-cased-sentence: 180M parameters, 12 layers, 768-dimensional vectors. The model is high quality but slow on a CPU. It has two known weaknesses.

The first is "blindness to negation". It distinguishes poorly between the phrases "got a job" and "did not get a job", even though their meanings are opposite. For deduplication and search this is a problem: the system will treat such texts as nearly identical.

The second is anisotropy. The vectors of all texts are squeezed into a narrow cone, so any two texts end up with high cosine similarity (a measure of how close two vectors are). It is hard to pick a threshold for "this is a duplicate": the deletion list ends up with tens of thousands of pairs instead of a handful. The base model sergeyzh/rubert-tiny-turbo, with its 312 dimensions and token averaging, compresses the space even more.

The constraints are strict: it must run on a CPU, take up to 300 MB, and run several times faster than the teacher. The task is not only to shrink the teacher but also to partly fix its shortcomings.

How the student is built

The student is based on rubert-tiny-turbo, a version of rubert-tiny2 pretrained contrastively (the model was trained to pull similar texts closer together and push dissimilar ones apart), so its semantics are better at the start. The tokenizer stays the original one, and the 84-thousand-token vocabulary is not expanded.

Two components are added on top of the encoder's ready-made hidden states:

  • Multi-head attention pooling with 4 heads instead of simple averaging. The model learns on its own to weight the words in a text: important ones, such as "not", get more weight. The outputs of the heads are averaged.
  • A linear projection 312 → 768. It is needed for training, so the student can be compared with the teacher in the teacher's space. After training, the projection can be kept or turned off: 312 dimensions give a smaller size and higher speed.

Whitening, that is, forced "smoothing" of the vector space, is not applied in the main model: the author expects contrastive training to bring the distribution closer to uniform on its own. For cases where the user's text differs strongly from the training data, an optional fit_domain() function is provided.

Training runs in two phases. In Phase A the encoder is frozen and only the new head is trained: it tries to make its vectors resemble the teacher's. In Phase B the top two layers are unfrozen, and two objectives are mixed in equal proportions: InfoNCE (telling the correct answer from the incorrect ones) and distillation (reproducing the teacher). The data consists of 243,412 texts from Wikipedia and a psychology corpus after deduplication, plus 99,999 triplets from deepvk/ru-HNP: a query, five matching texts and five "hard" non-matching ones.

What is useful here for practitioners

First, the goals are measurable in advance. Similarity to the teacher after Phase A should be at least 90%, and the final figure at least 85%. Retrieval accuracy on hard negative examples should be at least 97%, and turbo already starts at 96.8%. For deduplication the criterion is practical: only a handful of pairs should end up in auto-deletion, not tens of thousands. For clustering, the ratio of distances within clusters to distances between them should be clearly above one.

Second, the result is packaged as a working tool: a folder of about 120 MB (the weights alone are 117.8 MB) with a SemanticEmbedder class that offers encode, search, dedup, cluster and fit_domain. The student, turbo and the teacher are compared on the external ru-paraphrase-NMT-Leipzig set (5000 pairs) and on three practical tasks.

There are caveats too. The author states plainly that he is not a programmer: DeepSeek wrote all the code, and he did the debugging and error fixing. The texts were assembled from notes taken along the way, so they are rough in places. Quality figures will appear only in later parts, and they cannot be carried over to other domains without verification.

As a how-to guide, the series looks useful: it shows how to build a lightweight Russian-language model for CPUs without starting from scratch and without relying on a single metric.

Story

Key Questions Answered

What problem the project solves?

The teacher is DeepPavlov/rubert-base-cased-sentence: 180M parameters, 12 layers, 768-dimensional vectors. The model is high quality but slow on a CPU. It has two known weaknesses. The first is "blindness to negation". It distinguishes poorly between the phrases "got a job" and "did not get a job", even though their meanings are opposite. For deduplication and search this is a problem: the system will treat such texts as nearly identical. The second is anisotropy.

How the student is built?

The student is based on rubert-tiny-turbo, a version of rubert-tiny2 pretrained contrastively (the model was trained to pull similar texts closer together and push dissimilar ones apart), so its semantics are better at the start. The tokenizer stays the original one, and the 84-thousand-token vocabulary is not expanded. Two components are added on top of the encoder's ready-made hidden states: Multi-head attention pooling with 4 heads instead of simple averaging.

What is useful here for practitioners?

First, the goals are measurable in advance. Similarity to the teacher after Phase A should be at least 90%, and the final figure at least 85%. Retrieval accuracy on hard negative examples should be at least 97%, and turbo already starts at 96.8%. For deduplication the criterion is practical: only a handful of pairs should end up in auto-deletion, not tens of thousands. For clustering, the ratio of distances within clusters to distances between them should be clearly above one.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…