Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)

Zeroing Out Less Than 0.05% of Weights Taught a Large LLM New Skills as Well as Fine-Tuning

New skills are usually added by rewriting weights. Deletion Tuning instead zeroes out some of the existing ones: on the largest models, less than 0.05% of parameters was enough, which is fewer than 5 in 10,000. Only a "keep or zero out" mask for the MLP layers is trained. Quality matched full fine-tuning, and accuracy on old tasks dropped by no more than 1 point. That means a single model can be switched between skills using masks that take up megabytes. There are caveats. Smaller models sometimes reach only 90% of the accuracy of a dense update. Uncompressed bf16 rank-1 adapters are often smaller than the masks. Sequential fine-tuning was not tested, and training needs far more memory.

AI-processed from TG: @gonzo_ML (gonzo-обзоры ML статей); edited by Hamidun News
Zeroing Out Less Than 0.05% of Weights Taught a Large LLM New Skills as Well as Fine-Tuning
Source: TG: @gonzo_ML (gonzo-обзоры ML статей). Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceTG: @gonzo_ML (gonzo-обзоры ML статей) ↗
Editor-in-Chief
Key metric #1Small models sometimes reach only 90% of the accuracy of a dense update
Key metric #2Uncompressed bf16 rank-1 adapters are often smaller than the masks
Key metric #3Sequential fine-tuning was not tested, and much more memory is needed

Say you run one large language model in production, and your customers need different skills: one wants an agent that calls tools, another wants step-by-step reasoning. The usual approach is to rewrite all of the model's weights (full fine-tuning) or to bolt a small adapter such as LoRA on top. Researchers Jack Lu, Mengye Ren and their colleagues propose a third path, Deletion Tuning: add nothing, and instead zero out a tiny fraction of the weights that already exist.

On the largest models tested, zeroing out less than 0.05% of parameters was enough, which is fewer than five weights in ten thousand, and quality matched full fine-tuning. For anyone serving many variants of one model, this is an interesting engineering idea, though it is still far from a ready-to-use tool.

How to teach a model by only deleting

Imagine a big cookbook. Instead of adding pages, you strike out a few lines, and the cook starts cooking differently. The method works roughly like that. The original model stays frozen. Only a binary mask for the gate projections of the MLP blocks is trained. This is the part of the layer that decides which signals to pass on. For each weight, the mask says "keep" or "zero out." The weights that are kept are not changed at all.

This is how pretrained models pick up new reasoning and tool-use skills without any new weights being written. Their existing abilities do not suffer in the process.

What the measurements showed

At large scale, Deletion Tuning matched full fine-tuning, the most expensive way to adapt a model, in which all weights are updated. The authors also checked separately whether the model forgets what it knew before. After reinforcement learning training, accuracy on earlier tasks dropped by no more than 1 point relative to the base model. One point means, for example, a drop from 80% to 79%.

Results are worse with small models: they sometimes reach only about 90% of the accuracy of a dense update, meaning ordinary fine-tuning of all weights. This suggests the method works better on large networks. Perhaps it is easier to find weights that can be switched off without loss in them, but that is our guess, not a conclusion from the authors.

Why it matters in practice

The main benefit is in inference, that is, in serving a finished model. If a skill is stored as a bit mask that takes up megabytes, one shared model in GPU memory can be switched between tasks. One customer gets the "agent with tools" mask, another gets "reasoning." Old skills stay in place. The authors have released the code and the trained masks on Hugging Face, so you can test the idea yourself.

Caveats

First, the size savings are not guaranteed. Uncompressed rank-1 adapters in bf16 format often take up less space than the mask. So as a way to store many model variants more compactly than LoRA, the method does not yet look like an obvious winner.

Second, training the mask itself requires far more memory than training a small adapter. For teams with a limited GPU fleet, that is a serious drawback.

Third, the paper did not test sequential fine-tuning. That is the situation where a model gets one skill, then a second, then a third, which is how product teams usually work. It is unknown whether masks can be stacked on top of each other without conflicts.

Takeaway

Deletion Tuning remains a research result for now, not a tool you should drag into production tomorrow. But it shows that a large pretrained model already holds more capability than is commonly assumed, and that sometimes you can unlock it by zeroing out hundredths of a percent of the weights. If you are building multi-task serving on a single shared model, take a look at the paper and the code. If the training-memory problem and the sequential fine-tuning question can be solved, megabyte-sized masks could become a convenient alternative to adapters.

Story

Key Questions Answered

How to teach a model by only deleting?

Imagine a big cookbook. Instead of adding pages, you strike out a few lines, and the cook starts cooking differently. The method works roughly like that. The original model stays frozen. Only a binary mask for the gate projections of the MLP blocks is trained. This is the part of the layer that decides which signals to pass on. For each weight, the mask says "keep" or "zero out." The weights that are kept are not changed at all.

What the measurements showed?

At large scale, Deletion Tuning matched full fine-tuning, the most expensive way to adapt a model, in which all weights are updated. The authors also checked separately whether the model forgets what it knew before. After reinforcement learning training, accuracy on earlier tasks dropped by no more than 1 point relative to the base model. One point means, for example, a drop from 80% to 79%.

Why it matters in practice?

The main benefit is in inference, that is, in serving a finished model. If a skill is stored as a bit mask that takes up megabytes, one shared model in GPU memory can be switched between tasks. One customer gets the "agent with tools" mask, another gets "reasoning." Old skills stay in place. The authors have released the code and the trained masks on Hugging Face, so you can test the idea yourself.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Hands-On AI Engineering · Hamidun Academy

Build & Ship Production AI Agents Without Boilerplate Code

Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.

Start 72h Free Trial →Vibe-Coding Intensive
What do you think?
Loading comments…