NERVE: How to Mute the "SQL Neurons" in Qwen 27B and Fix 1C Queries Without Fine-Tuning

A familiar pain: you fine-tuned an open model on a narrow domain, yet it still regularly makes mistakes in the same place. New datasets barely help, and each round of data collection costs weeks. The author of an article on Habr ran into exactly this with Qwen 3.8 27B fine-tuned for 1C:Enterprise. The model writes code in the built-in language decently, but it confuses the 1C query language with SQL: it mixes up virtual tables, puts parameters in the wrong place, and drags in constructs that don't exist in 1C. Additional datasets gave less than 1% accuracy improvement.

The Hypothesis: SQL Is Interfering

The tokens of the two languages look alike: SELECT, JOIN, WITH. The author suggested that knowledge of real SQL pulls the model toward the wrong answer. In that case, instead of retraining, one can find the specific neurons responsible and weaken them. This is how the NERVE pipeline came about.

A neuron here means a single component of the intermediate vector of the MLP block (Qwen 3.8 27B has 17,408 of them per layer) together with the column of the W_down matrix it writes into. Multiplying the column by m means multiplying the neuron's activation by m on every token. A value of 0 switches the neuron off; 0.5 halves it. The architecture does not change, and the edit survives quantization and export to GGUF. In total there are about 1.1 million candidates (64 layers), so blind brute-force search is pointless.

How the Pipeline Works

1. Pairs of "correct / erroneous query" are taken, as for DPO, roughly two thousand of them. Since the answers match 90–95% of the time, only the differing tokens, found with difflib, are compared. Averaging over the whole answer would blur the signal.
2. Hooks on the input of each down_proj collect the mean activation of neurons on those tokens. Across all pairs, the mean, t-statistic, and sign consistency are computed. Ranking is by |t|, so that two or three anomalous pairs do not create false leaders.
3. Gradient attribution ("gradient × input") predicts how the margin, that is, how strongly the model prefers the correct answer over the erroneous one, will change if a neuron is muted. A neuron with a negative contribution is considered "erroneous". Attribution is expensive, so it is built on 512 pairs.
4. Greedy selection tries candidates one at a time on top of those already accepted. An edit is kept if the margin increased by at least 0.002 per token and the nll on correct answers dropped by no more than 2%. Since failed edits are rolled back, summed linear predictions are no longer needed.

The main safeguard against overfitting: the validation set is split into two halves by query hash. Decisions are made on one, and the result is checked on the other. A set is considered confirmed only if the margin increased on the control half and the nll did not worsen beyond tolerance. Confirmed edits accumulate in a JSON file next to the map and are excluded from future candidates.

What They Caught Along the Way

The first mistake: the map was built on differing tokens, but quality was measured over the whole answer. The effect was diluted roughly 25-fold. The fix: the same diff mask for both the map and the evaluation. The second finding: boosting "good" neurons more often broke nll, while suppressing erroneous ones gave a steady margin increase, so suppress mode became the primary one. One more observed fact: excitation on erroneous answers is present in almost every layer, meaning SQL-like behavior is smeared across the entire depth of the network.

Limitations and Takeaway

The available text contains no final numbers: we do not know how much accuracy rose on the control set. This is one model and one dataset, and the method relies on a linear approximation that breaks down when edits interact, which is why greedy selection is needed. For practitioners the idea is still useful: if a fine-tuned model has a persistent error of "leaking" from a neighboring domain, there is a cheap alternative to yet another round of SFT.

You need preference pairs, one forward-backward pass over a sample, and an honest control half. The key lesson: compute the metric on the same tokens used to build the map, and always verify on data that did not take part in selection.