Starts Nov 23Live Cohort 3 of Vibe Coding & AI Agents (Nov 23, 18:30 MSK): 3 weeks, 9 live hands-on sessionsCohort 3 (Nov 23) →
ReleaseInsight · On RadarHabr AI→ original

A strong Russian model rates "I love cats" and "I don't love cats" as 70% similar

An embedding turns a phrase into a set of numbers so that search can find texts with similar meaning. A Habr series author is choosing a base for a compact 29.4M-parameter Russian model and testing a large "teacher" from DeepPavlov. The teacher rates "I love cats" and "I don't love cats" as 70% similar, and "the doctor prescribed" and "the doctor cancelled the treatment" as 65.5% similar. For search and deduplication in medicine and law, test your model on negations in advance.

AI-processed from Habr AI; edited by Hamidun News
A strong Russian model rates "I love cats" and "I don't love cats" as 70% similar
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Turbo is almost twice as strong as tiny2a gap of 0.125 versus 0.064 with the same 3 layers
Key metric #2The 12-layer teacher is about 2.5 times stronger than turbo: 0.329 versus 0.125
Key metric #3The first run downloads about 940 MB, and the CPU check takes about 2 minutes

Imagine a search over a document base that treats "the doctor prescribed the treatment" and "the doctor cancelled the treatment" as almost identical. That is how a strong Russian embedding model behaves, the one the Habr series author took as a "teacher." An embedding turns text into a vector, a set of numbers, and the similarity of two phrases is computed as the cosine of the angle between the vectors: 100% (1.

0) means the meaning matches, and about 0% means nothing in common. The teacher scored "I love cats" and "I don't love cats" at 0.702, roughly 70% similarity, even though the meanings are opposite.

This is the first part of a ten-part series about compressing a model down to 29.4 million parameters.

What the author does and why

The method is called distillation: a small "student" learns to reproduce the behavior of a large "teacher" so it can run fast on an ordinary CPU, without a GPU. The author states upfront that he is not a programmer, that DeepSeek wrote all the code, and that his job was debugging and fixing errors. The environment is standard: PyTorch for CPU, transformers, sentence-transformers. The first run downloads about 940 MB (the DeepPavlov teacher is about 700 MB, turbo and tiny2 are 120 MB each), and the script itself finishes in two minutes.

There are three candidates. The first two, cointegrated/rubert-tiny2 and sergeyzh/rubert-tiny-turbo, each have 3 layers, 312 vector dimensions, and 29.4M parameters. The difference is that turbo was additionally trained on text comparison (contrastive training). The third, DeepPavlov/rubert-base-cased-sentence, has 12 layers and 768 dimensions and serves as the teacher.

How the models were compared

The metric is called the gap: the average similarity for pairs with the same meaning minus the average similarity for pairs with different meanings. The larger the gap, the better the model separates meaning. The test set is tiny, just 10 pairs: 4 paraphrases ("the cat sits on the chair" and "the cat sat down on the chair"), 3 pairs with a negation or an opposite action, and 3 pairs on unrelated topics like borscht and the Pythagorean theorem.

The author stresses that this is exploratory work for choosing a layer and a method for merging tokens into one vector (mean, CLS, max, last-token), not a quality assessment. He plans to measure quality later on 590 validation pairs and an external set of 5,000 pairs.

The best combinations were:

  • tiny2: layer L2 with CLS, gap 0.064;
  • turbo: layer L2 with token mean, gap 0.125;
  • DeepPavlov: layer L11 with mean, gap 0.329.

Turbo is almost twice as good as tiny2 with the same architecture: pretraining on text comparison helped, whereas tiny2 remains a base model trained to guess missing words. The teacher is about 2.5 times stronger than turbo. The author does not aim to match it on absolute semantics. It is enough to come close enough for search and duplicate removal to work.

Blindness to negation

The main finding is not the size but the fact that the teacher barely notices the word "not." Similarity on pairs with opposite meanings:

  • "he got a job" and "he did not get a job": 0.578 (about 58%);
  • "I love cats" and "I don't love cats": 0.702 (about 70%);
  • "the doctor prescribed the treatment" and "the doctor cancelled the treatment": 0.655 (about 66%).

The author explains this as follows. The base model was trained to restore missing words, so the contribution of one short word gets diluted. Averaging over tokens weakens it further. In practice, a deduplication threshold fires falsely: records with opposite meanings count as duplicates. Search then ranks the "opposite" results at the top. For psychotherapy, legal work, and medicine this is dangerous: "the drug was prescribed" and "the drug was discontinued" must not be merged.

The author chose turbo as the student. Its gap is already higher, so the semantics need to be adapted rather than learned from scratch. Only the encoder is taken; the built-in averaging is replaced with a custom attention pooling mechanism, and the representations learned from comparison are kept. The author plans to suppress negations through contrastive training on a set of hard pairs.

What to take away

Before choosing an embedder, assemble your own mini-set of 10–20 pairs: negations, antonyms, and verb substitutions. This takes about an hour and surfaces the problem before production. Do not assume a bigger model is automatically better: the blind spot for "not" may well be there too.

The limitations are obvious. Ten pairs are too few to draw conclusions about quality, and the author acknowledges this himself. The figures apply to his set and are not guaranteed to repeat on your data. The result of distillation will only appear in later parts, so it is too early to judge it. But the method of choosing a base is transparent, and the negation check will be useful in any project with semantic search.

Story

Key Questions Answered

What the author does and why?

The method is called distillation: a small "student" learns to reproduce the behavior of a large "teacher" so it can run fast on an ordinary CPU, without a GPU. The author states upfront that he is not a programmer, that DeepSeek wrote all the code, and that his job was debugging and fixing errors. The environment is standard: PyTorch for CPU, transformers, sentence-transformers.

How the models were compared?

The metric is called the gap: the average similarity for pairs with the same meaning minus the average similarity for pairs with different meanings. The larger the gap, the better the model separates meaning. The test set is tiny, just 10 pairs: 4 paraphrases ("the cat sits on the chair" and "the cat sat down on the chair"), 3 pairs with a negation or an opposite action, and 3 pairs on unrelated topics like borscht and the Pythagorean theorem.

What are the key details on Blindness to negation?

The main finding is not the size but the fact that the teacher barely notices the word "not." Similarity on pairs with opposite meanings: "he got a job" and "he did not get a job": 0.578 (about 58%); "I love cats" and "I don't love cats": 0.702 (about 70%); "the doctor prescribed the treatment" and "the doctor cancelled the treatment": 0.655 (about 66%). The author explains this as follows. The base model was trained to restore missing words, so the contribution of one short word gets diluted.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…