Concertina folded Qwen3-4B like an accordion: 43% smaller, with 84% of the original's quality
The authors did not throw away Qwen3-4B's layers. They folded them in pairs, like an accordion: one shared block that runs twice replaces every two adjacent layers. The parameter count drops from 4.02 billion to 2.27 billion (43% fewer), and quality is about 84% of the original model. Fine-tuning took 22 billion tokens, and the model beats Qwen3-1.7B, which was trained on 36 trillion tokens, on six of eight benchmarks. So far the savings are in memory only: speed was not measured.
AI-processed from Habr AI; edited by Hamidun News
There are usually two ways to get a small language model. You either train it from scratch on a huge corpus, or you cut some layers out of a large model and hope it doesn't get dumber. The first costs a fortune; the second hurts quality.
Researchers have now proposed a third route called Concertina: instead of discarding the layers of Qwen3-4B, they fold them in pairs, like the bellows of a concertina. The model shrank from 4.02 billion to 2.
27 billion parameters (by 43%) and kept about 84% of the original Qwen3-4B's quality (an average over eight benchmarks: the compressed model's score divided by the teacher's score). It needed only 22 billion tokens of distillation to get there. If a model doesn't fit in the memory of your laptop, phone or budget GPU, the result is worth a closer look.
How to fold a model like an accordion
The authors of the paper "Folding Language Models into Themselves" started with Qwen3-4B. They replaced every pair of adjacent layers with a single shared block that runs twice in a row. The shared matrices come from the first layer of the pair, and the network's depth stays the same: 36 layers, that is, 18 unique blocks with two passes each.
Imagine two neighboring employees doing almost the same job. Instead of paying for two positions, you hire one person and ask them to do the task twice, with a small correction the second time. The authors compute that correction from the difference between the original layers. They take a compressed rank-64 approximation of it (truncated SVD), and a per-token gate decides how strongly to apply it. Each pass keeps its own RMSNorm normalization.
After assembly, the resulting "student" model is distilled from the same Qwen3-4B, meaning it is trained to reproduce the original model's outputs. This took 22 billion tokens.
What the benchmarks show
The comparison covered the Qwen3-4B teacher, Qwen3-1.7B (trained on 36 trillion tokens, roughly 1,600 times more data) and Concertina-2.27B.
Concertina beats Qwen3-1.7B on six of eight metrics: the average score over the seven base benchmarks, base-7 (58.0% vs 56.2%), MMLU-Pro (30.1% vs 29.7%), GPQA with graduate-level questions (30.8% vs 29.3%), BBH for hard reasoning (40.0% vs 36.3%), GSM8K grade-school math (78.8% vs 71.0%) and MATH (49.8% vs 46.0%). It trails by a hair on MMLU (55.6% vs 55.7%) and MMLU-Redux (58.7% vs 59.1%). It does not catch the Qwen3-4B teacher on any benchmark: for example, the teacher scores 63.0% on base-7 and 87.9% on GSM8K.
By our calculation from the table, the average share of the teacher's quality is 84%, as the authors state. Reasoning tasks hold up best: about 90% on base-7 and GSM8K. Knowledge and hard questions lose more: MMLU-Pro and BBH sit at 75–80% of the teacher.
What you save and what you don't
The model's bf16 weights shrink from 8.0 to 4.5 GB. For devices where weight memory is the bottleneck, the gain is immediate: a model that used to not fit now does.
The rest is less clear. The effective depth is still 36 layers, so the KV-cache is the same as in Qwen3-4B. Compute per token is probably about the same too, since every shared block runs twice. The model card has no speed or throughput measurements, so there is nothing yet to confirm a benefit for server-side inference, where speed and cache size decide.
The release itself also comes with caveats. The repository has two commits and a single checkpoint: there are no other layer-group sizes or other correction ranks. The method is demonstrated on one model, and how it transfers to other families is still unknown. The code is short: the fold_model and distill_loss functions and a self-test on CPU. The weights are on Hugging Face under the Apache 2.0 license, like the original Qwen3-4B.
What to do with this
If you deploy models on edge devices or are short on VRAM, the checkpoint is worth downloading and running on your own tasks, especially those that need reasoning and math rather than encyclopedic knowledge. The main thing to measure yourself is real tokens/sec on your GPU compared with the standard Qwen3-4B. If speed doesn't improve, Concertina remains a way to make a model smaller in memory, but not faster. The idea is still instructive: 22 billion distillation tokens against 36 trillion tokens of full training suggest that compressing an already trained model can be noticeably cheaper than growing a small one from scratch.
- Nov 12, 2026AI agents re-checked 2,000+ ICML 2026 papers and found errors that reviewers missed
- Nov 12, 2026Eight Countries, OpenAI and Anthropic Promise an AI "Kill Switch" — but Set No Deadlines or Penalties
- Nov 12, 2026Hugging Face warns: a wave of hacks could make open-source AI less open
- Nov 12, 2026OpenAI Built a Control Room: A Kill Switch for Models That Escaped the Sandbox
Key Questions Answered
How to fold a model like an accordion?
The authors of the paper "Folding Language Models into Themselves" started with Qwen3-4B. They replaced every pair of adjacent layers with a single shared block that runs twice in a row. The shared matrices come from the first layer of the pair, and the network's depth stays the same: 36 layers, that is, 18 unique blocks with two passes each. Imagine two neighboring employees doing almost the same job.
What the benchmarks show?
The comparison covered the Qwen3-4B teacher, Qwen3-1.7B (trained on 36 trillion tokens, roughly 1,600 times more data) and Concertina-2.27B. Concertina beats Qwen3-1.7B on six of eight metrics: the average score over the seven base benchmarks, base-7 (58.0% vs 56.2%), MMLU-Pro (30.1% vs 29.7%), GPQA with graduate-level questions (30.8% vs 29.3%), BBH for hard reasoning (40.0% vs 36.3%), GSM8K grade-school math (78.8% vs 71.0%) and MATH (49.8% vs 46.0%).
What you save and what you don't?
The model's bf16 weights shrink from 8.0 to 4.5 GB. For devices where weight memory is the bottleneck, the gain is immediate: a model that used to not fit now does. The rest is less clear. The effective depth is still 36 layers, so the KV-cache is the same as in Qwen3-4B. Compute per token is probably about the same too, since every shared block runs twice.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.