Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

kazRush Beats NLLB 3.3B on BLEU With a Model 17 Times Smaller

The VK team assembled an open Russian-Kazakh corpus, cleaned it down to 9 million pairs, and trained kazRush from scratch. The 197-million-parameter model beat NLLB 3.3B on BLEU and chrF for translation into Kazakh, although it lagged on COMET. The practical lesson extends beyond translation: data provenance, careful filtering, and a task-specific architecture can deliver more than mechanical scaling.

AI-processed from Habr AI; edited by Hamidun News
kazRush Beats NLLB 3.3B on BLEU With a Model 17 Times Smaller
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Key metric #1Near-duplicates proved useful as a form of natural augmentation.
Key metric #2An 8,000-token vocabulary performed better than the 16,000-token version.
Key metric #3Scaling the model to 738 million parameters did not improve translation.

In machine translation, it is easy to fall into an expensive trap: collect more data, choose a larger model, and wait for quality to improve on its own. VK's experimental AI technologies team tested this recipe on Russian-Kazakh translation and reached a more useful conclusion. The 197-million-parameter kazRush model beat NLLB 3.3B on BLEU and chrF when translating from Russian into Kazakh. Scale alone was not the answer; what worked was a combination of a carefully prepared corpus, a compact tokenizer, and an architecture matched to the volume of data.

Data First, Parameters Second

The initial corpus was assembled from OPUS, KazParC, WMT, and Turkic Interlingua. Before filtering, it contained about 12.3 million sentence pairs. After removing noise, checking languages, and aligning the data, 9,054,960 pairs remained.

A simple example illustrates the problem. One row may contain a Russian sentence on the left and, on the right, a Kazakh passage that discusses a similar topic but is not its translation. Formally, both texts are written in the required languages, so a basic check will let the pair through. The model, however, will learn a false correspondence.

The team systematically removed exact duplicates, HTML markup, and stray characters; applied Unicode normalization; identified the language of every sentence with fastText; and measured semantic similarity using LaBSE. Pairs scoring below 0.4 were discarded. The team also checked numbers, length ratios, and the share of letters and digits in each line. Some sources required custom rules: a universal filter was not enough.

The search for near-duplicates produced an intriguing result. Similar lines might appear to do nothing but pollute the corpus. Yet removing them reduced model quality. One possible explanation is that, when data is scarce, small variations of the same expression act as natural augmentation. The final dataset therefore excluded exact repetitions but retained near-duplicates.

A Compact Model Instead of a Size Race

The team trained a shared SentencePiece tokenizer for the two languages with 8,000 subword units. A 16,000-unit version performed slightly worse. This is an important reminder: a larger vocabulary is not always beneficial, especially for a morphologically rich language and a limited corpus.

The model was based on an encoder-decoder Transformer in the style of T5, but without pretrained weights. The final configuration had 12 layers in both the encoder and decoder, a hidden-state size of 512, and feed-forward blocks of size 4,096. In total, it contained about 197 million parameters.

The authors compared it with T5-Small, T5-Base, and T5-Large. The 738-million-parameter model was not the best: for Russian-to-Kazakh translation, it scored 15.8 BLEU, while kazRush reached 16.2. On chrF, the scores were 51.9 versus 51.6, and on COMET, 0.8836 versus 0.8804. In other words, an almost fourfold increase in parameter count did not pay off.

Where the Result Is Genuinely Strong

On the FLORES+ devtest set, kazRush was compared with the NLLB family. For Russian-to-Kazakh translation, the model scored 16.2 BLEU and 51.8 chrF, beating NLLB 3.3B at 15.6 and 50.7. On COMET, however, the larger NLLB remained ahead: 0.889 versus 0.883. It would therefore be wrong to claim an unconditional victory, because different metrics evaluate different properties of a translation.

In the reverse direction, kazRush also beat NLLB distilled 600M but trailed the NLLB versions with 1.3 billion and 3.3 billion parameters. It is a specialized tool, not a universal replacement for large multilingual models.

The practical conclusion applies far beyond translation. Preserve the provenance of every record, measure the effect of each filter, and do not remove data merely because a rule appears reasonable. First run ablations on the corpus, tokenizer, and architecture—and only then pay for additional compute. Sometimes the best way to make a model smaller is to take a closer look at what you are feeding it.

Story

Key Questions Answered

What are the key details on Data First, Parameters Second?

The initial corpus was assembled from OPUS, KazParC, WMT, and Turkic Interlingua. Before filtering, it contained about 12.3 million sentence pairs. After removing noise, checking languages, and aligning the data, 9,054,960 pairs remained. A simple example illustrates the problem. One row may contain a Russian sentence on the left and, on the right, a Kazakh passage that discusses a similar topic but is not its translation.

What are the key details on A Compact Model Instead of a Size Race?

The team trained a shared SentencePiece tokenizer for the two languages with 8,000 subword units. A 16,000-unit version performed slightly worse. This is an important reminder: a larger vocabulary is not always beneficial, especially for a morphologically rich language and a limited corpus. The model was based on an encoder-decoder Transformer in the style of T5, but without pretrained weights.

Where the Result Is Genuinely Strong?

On the FLORES+ devtest set, kazRush was compared with the NLLB family. For Russian-to-Kazakh translation, the model scored 16.2 BLEU and 51.8 chrF, beating NLLB 3.3B at 15.6 and 50.7. On COMET, however, the larger NLLB remained ahead: 0.889 versus 0.883. It would therefore be wrong to claim an unconditional victory, because different metrics evaluate different properties of a translation.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…