Zeroing Out Less Than 0.05% of Weights Taught a Large LLM New Skills as Well as Fine-Tuning
New skills are usually added by rewriting weights. Deletion Tuning instead zeroes out some of the existing ones: on the largest models, less than 0.05% of parameters was enough, which is fewer than 5 in 10,000. Only a "keep or zero out" mask for the MLP layers is trained. Quality matched full fine-tuning, and accuracy on old tasks dropped by no more than 1 point. That means a single model can be switched between skills using masks that take up megabytes. There are caveats. Smaller models sometimes reach only 90% of the accuracy of a dense update. Uncompressed bf16 rank-1 adapters are often smaller than the masks. Sequential fine-tuning was not tested, and training needs far more memory.
AI-processed from TG: @gonzo_ML (gonzo-обзоры ML статей); edited by Hamidun News
Say you run one large language model in production, and your customers need different skills: one wants an agent that calls tools, another wants step-by-step reasoning. The usual approach is to rewrite all of the model's weights (full fine-tuning) or to bolt a small adapter such as LoRA on top. Researchers Jack Lu, Mengye Ren and their colleagues propose a third path, Deletion Tuning: add nothing, and instead zero out a tiny fraction of the weights that already exist.
On the largest models tested, zeroing out less than 0.05% of parameters was enough, which is fewer than five weights in ten thousand, and quality matched full fine-tuning. For anyone serving many variants of one model, this is an interesting engineering idea, though it is still far from a ready-to-use tool.
How to teach a model by only deleting
Imagine a big cookbook. Instead of adding pages, you strike out a few lines, and the cook starts cooking differently. The method works roughly like that. The original model stays frozen. Only a binary mask for the gate projections of the MLP blocks is trained. This is the part of the layer that decides which signals to pass on. For each weight, the mask says "keep" or "zero out." The weights that are kept are not changed at all.
This is how pretrained models pick up new reasoning and tool-use skills without any new weights being written. Their existing abilities do not suffer in the process.
What the measurements showed
At large scale, Deletion Tuning matched full fine-tuning, the most expensive way to adapt a model, in which all weights are updated. The authors also checked separately whether the model forgets what it knew before. After reinforcement learning training, accuracy on earlier tasks dropped by no more than 1 point relative to the base model. One point means, for example, a drop from 80% to 79%.
Results are worse with small models: they sometimes reach only about 90% of the accuracy of a dense update, meaning ordinary fine-tuning of all weights. This suggests the method works better on large networks. Perhaps it is easier to find weights that can be switched off without loss in them, but that is our guess, not a conclusion from the authors.
Why it matters in practice
The main benefit is in inference, that is, in serving a finished model. If a skill is stored as a bit mask that takes up megabytes, one shared model in GPU memory can be switched between tasks. One customer gets the "agent with tools" mask, another gets "reasoning." Old skills stay in place. The authors have released the code and the trained masks on Hugging Face, so you can test the idea yourself.
Caveats
First, the size savings are not guaranteed. Uncompressed rank-1 adapters in bf16 format often take up less space than the mask. So as a way to store many model variants more compactly than LoRA, the method does not yet look like an obvious winner.
Second, training the mask itself requires far more memory than training a small adapter. For teams with a limited GPU fleet, that is a serious drawback.
Third, the paper did not test sequential fine-tuning. That is the situation where a model gets one skill, then a second, then a third, which is how product teams usually work. It is unknown whether masks can be stacked on top of each other without conflicts.
Takeaway
Deletion Tuning remains a research result for now, not a tool you should drag into production tomorrow. But it shows that a large pretrained model already holds more capability than is commonly assumed, and that sometimes you can unlock it by zeroing out hundredths of a percent of the weights. If you are building multi-task serving on a single shared model, take a look at the paper and the code. If the training-memory problem and the sequential fine-tuning question can be solved, megabyte-sized masks could become a convenient alternative to adapters.
- Nov 18, 2026Token position recovered at 99% from a NoPE transformer, but only 29–42% from RWKV-7
- Nov 18, 2026Liquid AI Releases LiquidCodec: Decodes Speech 2.9x Faster Than Mimi
- Nov 18, 20261,000 AI agents wrote a chatbot with no neural network: win rate rose from 0% to 26%
- Nov 18, 2026Concertina folded Qwen3-4B like an accordion: 43% smaller, with 84% of the original's quality
Key Questions Answered
How to teach a model by only deleting?
Imagine a big cookbook. Instead of adding pages, you strike out a few lines, and the cook starts cooking differently. The method works roughly like that. The original model stays frozen. Only a binary mask for the gate projections of the MLP blocks is trained. This is the part of the layer that decides which signals to pass on. For each weight, the mask says "keep" or "zero out." The weights that are kept are not changed at all.
What the measurements showed?
At large scale, Deletion Tuning matched full fine-tuning, the most expensive way to adapt a model, in which all weights are updated. The authors also checked separately whether the model forgets what it knew before. After reinforcement learning training, accuracy on earlier tasks dropped by no more than 1 point relative to the base model. One point means, for example, a drop from 80% to 79%.
Why it matters in practice?
The main benefit is in inference, that is, in serving a finished model. If a skill is stored as a bit mask that takes up megabytes, one shared model in GPU memory can be switched between tasks. One customer gets the "agent with tools" mask, another gets "reasoning." Old skills stay in place. The authors have released the code and the trained masks on Hugging Face, so you can test the idea yourself.
Build & Ship Production AI Agents Without Boilerplate Code
Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.