SFT-as-Context Restores General Skills in Fine-Tuned LLMs, Within 2.2 Points

Fine-tuning a language model almost always costs it some of its earlier skills. The authors of an arXiv preprint propose a way around that cost without a single gradient descent step. The method, SFT-as-context, has the original parent model answer a query while holding the response from its SFT-fine-tuned version in context. Across 19 "parent — SFT model" pairs and 11 benchmarks, the gap to the SFT model on specialized tasks was about two percentage points. On general skills, results stayed within 2.2 percentage points of the parent on average.

The problem: specialization at the cost of forgetting

Supervised fine-tuning turns a general-purpose model into a specialist in math, code, or another narrow field. The side effect is well known: the model loses some of the parent's general abilities. Queries that need both skill sets at once suffer the most. A fine-tuned model may confidently solve the domain-specific part and fail the rest. The parent will do the opposite.

This problem is usually addressed by mixing general data into training, merging weights, or using more conservative fine-tuning schemes. All of these require compute and careful tuning. The authors take a different route.

How SFT-as-context works

The scheme has two steps. First, the SFT model receives the query and produces its answer. Then the parent model receives the same query with that answer in its context and writes the final response. The weights are not touched. The parent picks up specialized techniques through in-context learning while keeping its own general abilities.

A side effect is that every query requires two passes through the models and a longer context. The abstract gives no figures for latency or cost, so you will have to estimate them yourself.

What the tests showed

The authors compared the method against both models in each pair: against the SFT version on specialized tasks, and against the parent on general ones. Key figures from the abstract:

• 19 "parent — SFT model" pairs and 11 benchmarks.
• AIME 2024: a 2.2 percentage point gap to the SFT model.
• LiveCodeBench: a 2.1 percentage point gap to the SFT model.
• NutriBench-English: a difference of 2.0 in macro MAE.
• General skills: within 2.2 percentage points of the parent model on average.

The most interesting result concerns tasks at the intersection of skills. The method solved queries that require both specialized and general abilities, even when neither the parent nor the SFT model could handle them alone. The specialist's answer works as a hint, and the parent fills in the missing pieces with its own knowledge.

The approach is not limited to "related" pairs. Answers from a small open SFT model improved the results of a strong closed LLM, and the combination outperformed each model on its own. For teams that work through an API and cannot change weights, this is the most practical takeaway of the paper.

Theory and attention

The authors built a Bayesian model and derived error bounds. On specialized tasks, the method's error is bounded relative to the SFT model; on general tasks, relative to the parent. These are guarantees within a theoretical framework, not a promise for every real-world scenario.

They also examined the mechanism. According to attention maps, the parent model pays more attention to useful SFT answers and less to irrelevant ones. This looks like selective use of the hint. It does not guarantee protection against a wrong hint, and the abstract does not say how the method behaves on clearly incorrect SFT answers.

Where this is useful in practice

If you have a fine-tuned model that falls short on general tasks, SFT-as-context is worth testing as a cheap baseline before expensive weight-merging experiments. A script with two calls is enough. Products built on closed models gain another option: a small domain model on your own hardware produces a draft answer, and a strong cloud model refines it into the final result.

Before deploying, measure three things: the quality gain on your data, the doubled latency and token consumption, and the share of cases where a wrong SFT answer drags the final result down. The abstract of the original paper does not disclose these figures, so you will have to decide based on your own measurements.