A strong Russian model rates "I love cats" and "I don't love cats" as 70% similar
An embedding turns a phrase into a set of numbers so that search can find texts with similar meaning. A Habr series author is choosing a base for a compact 29.4M-parameter Russian model and testing a large "teacher" from DeepPavlov. The teacher rates "I love cats" and "I don't love cats" as 70% similar, and "the doctor prescribed" and "the doctor cancelled the treatment" as 65.5% similar. For search and deduplication in medicine and law, test your model on negations in advance.
AI-processed from Habr AI; edited by Hamidun News
Imagine a search over a document base that treats "the doctor prescribed the treatment" and "the doctor cancelled the treatment" as almost identical. That is how a strong Russian embedding model behaves, the one the Habr series author took as a "teacher." An embedding turns text into a vector, a set of numbers, and the similarity of two phrases is computed as the cosine of the angle between the vectors: 100% (1.
0) means the meaning matches, and about 0% means nothing in common. The teacher scored "I love cats" and "I don't love cats" at 0.702, roughly 70% similarity, even though the meanings are opposite.
This is the first part of a ten-part series about compressing a model down to 29.4 million parameters.
What the author does and why
The method is called distillation: a small "student" learns to reproduce the behavior of a large "teacher" so it can run fast on an ordinary CPU, without a GPU. The author states upfront that he is not a programmer, that DeepSeek wrote all the code, and that his job was debugging and fixing errors. The environment is standard: PyTorch for CPU, transformers, sentence-transformers. The first run downloads about 940 MB (the DeepPavlov teacher is about 700 MB, turbo and tiny2 are 120 MB each), and the script itself finishes in two minutes.
There are three candidates. The first two, cointegrated/rubert-tiny2 and sergeyzh/rubert-tiny-turbo, each have 3 layers, 312 vector dimensions, and 29.4M parameters. The difference is that turbo was additionally trained on text comparison (contrastive training). The third, DeepPavlov/rubert-base-cased-sentence, has 12 layers and 768 dimensions and serves as the teacher.
How the models were compared
The metric is called the gap: the average similarity for pairs with the same meaning minus the average similarity for pairs with different meanings. The larger the gap, the better the model separates meaning. The test set is tiny, just 10 pairs: 4 paraphrases ("the cat sits on the chair" and "the cat sat down on the chair"), 3 pairs with a negation or an opposite action, and 3 pairs on unrelated topics like borscht and the Pythagorean theorem.
The author stresses that this is exploratory work for choosing a layer and a method for merging tokens into one vector (mean, CLS, max, last-token), not a quality assessment. He plans to measure quality later on 590 validation pairs and an external set of 5,000 pairs.
The best combinations were:
- tiny2: layer L2 with CLS, gap 0.064;
- turbo: layer L2 with token mean, gap 0.125;
- DeepPavlov: layer L11 with mean, gap 0.329.
Turbo is almost twice as good as tiny2 with the same architecture: pretraining on text comparison helped, whereas tiny2 remains a base model trained to guess missing words. The teacher is about 2.5 times stronger than turbo. The author does not aim to match it on absolute semantics. It is enough to come close enough for search and duplicate removal to work.
Blindness to negation
The main finding is not the size but the fact that the teacher barely notices the word "not." Similarity on pairs with opposite meanings:
- "he got a job" and "he did not get a job": 0.578 (about 58%);
- "I love cats" and "I don't love cats": 0.702 (about 70%);
- "the doctor prescribed the treatment" and "the doctor cancelled the treatment": 0.655 (about 66%).
The author explains this as follows. The base model was trained to restore missing words, so the contribution of one short word gets diluted. Averaging over tokens weakens it further. In practice, a deduplication threshold fires falsely: records with opposite meanings count as duplicates. Search then ranks the "opposite" results at the top. For psychotherapy, legal work, and medicine this is dangerous: "the drug was prescribed" and "the drug was discontinued" must not be merged.
The author chose turbo as the student. Its gap is already higher, so the semantics need to be adapted rather than learned from scratch. Only the encoder is taken; the built-in averaging is replaced with a custom attention pooling mechanism, and the representations learned from comparison are kept. The author plans to suppress negations through contrastive training on a set of hard pairs.
What to take away
Before choosing an embedder, assemble your own mini-set of 10–20 pairs: negations, antonyms, and verb substitutions. This takes about an hour and surfaces the problem before production. Do not assume a bigger model is automatically better: the blind spot for "not" may well be there too.
The limitations are obvious. Ten pairs are too few to draw conclusions about quality, and the author acknowledges this himself. The figures apply to his set and are not guaranteed to repeat on your data. The result of distillation will only appear in later parts, so it is too early to judge it. But the method of choosing a base is transparent, and the negation check will be useful in any project with semantic search.
- Nov 22, 2026DeepSeek Veteran Open-Sources Linger: A Local AI Agent That Responds Up to 5x Faster
- Nov 22, 2026Linger Runs an AI Agent on a 16 GB Mac With No Token Fees and Promises a 5x Speedup
- Nov 22, 2026Yandex Grant Funds an LLM Engine: 35B Model on RTX 2080 Ti Runs Three Times Faster Than llama.cpp
- Nov 22, 2026Two AI Judges, 15 LLMs: Judges Favor Their Own Makers by 0.04 Points Out of 1
Key Questions Answered
What the author does and why?
The method is called distillation: a small "student" learns to reproduce the behavior of a large "teacher" so it can run fast on an ordinary CPU, without a GPU. The author states upfront that he is not a programmer, that DeepSeek wrote all the code, and that his job was debugging and fixing errors. The environment is standard: PyTorch for CPU, transformers, sentence-transformers.
How the models were compared?
The metric is called the gap: the average similarity for pairs with the same meaning minus the average similarity for pairs with different meanings. The larger the gap, the better the model separates meaning. The test set is tiny, just 10 pairs: 4 paraphrases ("the cat sits on the chair" and "the cat sat down on the chair"), 3 pairs with a negation or an opposite action, and 3 pairs on unrelated topics like borscht and the Pythagorean theorem.
What are the key details on Blindness to negation?
The main finding is not the size but the fact that the teacher barely notices the word "not." Similarity on pairs with opposite meanings: "he got a job" and "he did not get a job": 0.578 (about 58%); "I love cats" and "I don't love cats": 0.702 (about 70%); "the doctor prescribed the treatment" and "the doctor cancelled the treatment": 0.655 (about 66%). The author explains this as follows. The base model was trained to restore missing words, so the contribution of one short word gets diluted.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.