HyperRepair Restores 93.4% of General Skills to Fine-Tuned LLMs While Retaining 99.5% of Domain Score

A team from Shanghai Jiao Tong University and the University of Queensland has described HyperRepair, a method that restores the general skills a language model loses when it is adapted to a narrow task. It needs neither target-domain data nor any extra training on the user's side. A small side network plugs into the frozen model, and a separate hypernetwork generates that network's weights. Across 12 LoRA datasets and 15 general benchmarks, the method recovered 93.4% of the base models' average general score while retaining 99.5% of the fine-tuned versions' average domain score.

Why post-fine-tuning repair runs into a trade-off

Fine-tuning improves a model in its target area but weakens its general knowledge and reasoning. Even LoRA, which trains only a small fraction of parameters, does not protect against catastrophic forgetting. Existing countermeasures fall into two camps. One group intervenes in the fine-tuning process itself: parameter regularization, replay of old data, distillation, restructured adapters and loss weighting. These approaches require control over training and do not help when the checkpoint is already finished. The other group works after the fact, by merging models or pruning weights with forgetting in mind. But a single set of modified weights applies to every query, and gains in general skills are usually paid for with a loss of domain specialization.

The authors took a different route. The fine-tuned model's weights stay untouched, and the correction lives in a separate channel. The original model is kept as a fallback, and the prompt itself can be used to decide whether the correction is needed.

A gated side network and the hypernetwork that generates it

The Repairing Side Net (RSN) is a lightweight transformer network attached to the intermediate states of the frozen model. It outputs a residual correction to the final hidden state before the language head. A gate decides how much of the correction to apply: it averages the states over the prompt's tokens and passes them through a small MLP with a sigmoid. The gate is trained to output one on general prompts and zero on unlabeled domain prompts. At zero, the model produces the same predictions as the fine-tuned version.

The most interesting part is how an RSN is obtained for a new model without training. Each fine-tuned model is described by an "activation fingerprint": on a fixed set of unlabeled general prompts, the authors measure how each layer's hidden states have shifted relative to the pretrained model. The shift is decomposed into a direction and the logarithm of its magnitude. A transformer encodes these features layer by layer and builds local and global representations. From these, the HyperNet generates the RSN parameters together with the gate: a shared learnable template plus a fingerprint-conditioned addition. Matrix corrections are assembled from low-rank factors, which reduces the number of values to generate.

First, a dedicated RSN is trained for each source model, and its parameters become the targets for the hypernetwork. The hypernetwork is trained with three signals: the mean squared error between generated and reference weights, alignment of the features the gate reads, and a functional loss on the predictions of the model with the generated RSN attached. The functional term is needed because errors in the weights affect both the correction itself and the gate's decision. At deployment, for a model from an unknown domain, the system computes the fingerprint, runs the hypernetwork once and attaches the finished RSN.

What the tests showed and where questions remain

The authors evaluated the method on two model families, 12 adaptation domains (classification, question answering, semantic parsing, code summarization) and 15 general benchmarks. The base checkpoints named include Qwen2.5-7B; the description text cuts off, so the second family and the dataset details are not visible here. The hypernetwork was trained on source models and tested on domains that were absent from training. On the standard split, the result is 93.4% of the base models' general score and 99.5% of the fine-tuned models' domain score.

The practical value is clear: if a company keeps many LoRA adapters on one base model, repairing each with separate training scales linearly with their number, whereas here the cost comes down to one pass through the hypernetwork. The limitations are just as clear. The method is tied to a shared pretrained base, the correction adds compute at inference, and 93.4% means the recovery is not complete. The available text does not say whether the approach works at other model sizes and on other architectures, or how the gate behaves on borderline prompts. Independent replication will have to test that.