Near-Zero Token Embeddings Wiped Out 20% of Fine-Tuning Steps in Gated Models on Telugu

If you fine-tune an open gated model — Mamba, Gated DeltaNet or xLSTM, that is, a recurrent architecture in which "gates" regulate access to the accumulated state — a noticeable share of optimization steps may not change the weights at all. The cause is not a data error but tokens whose input embeddings have a near-zero norm. In such gated models, a single such token amplifies gradients by dozens of orders of magnitude, and standard gradient clipping turns that explosion into a complete zeroing of the update. In continual pretraining on Telugu, this wastes roughly 20% of steps, and the logs show neither exceptions nor warnings.

Where the "zero" rows in checkpoints come from

The authors checked 291 open Hugging Face repositories (280 distinct embedding tables). They read the weights and tokenizer directly through HTTP byte-range requests. To filter out special and unreachable entries, they ran every suspicious row through a check: they decoded the ID and re-tokenized it, both on its own and in eight phrase templates. A token counted as reachable only if it produced the same ID.

Of the 291 repositories, 92 contain rows with a very small norm, and 67 (23.0%) contain reachable near-zero tokens. Among the 101 models with tied input and output matrices, there is not a single such token. Among the 190 models with separate matrices, 67 (35.3%) have them. For standard Transformers the share is 30.9%; for gated and hybrid models it is 52.6% (20 of 38).

The cause lies in pretraining. A token that never appeared in the text receives no gradient in its input row. But standard AdamW applies weight decay to all two-dimensional matrices, so an unused row shrinks toward zero step after step.

The output row, meanwhile, receives nonzero softmax gradients on every step and stays healthy: for 92% of 979 checked tokens (901), output norms are at or above the vocabulary median. With tied weights, the output gradients themselves keep the row at a normal scale. In a sample of 364 million tokens, all 251 near-zero rows belong to tokens with zero occurrences.

Ordinary rows are affected: Telugu and Basque subwords, Mongolian words in Cyrillic, code fragments with CRLF. Fine-tuned and distilled models inherit these rows from their base checkpoints.

Why gated models suffer in particular

In a Transformer, a token at any position other than the first is not a problem. The very first attention layer averages the values of preceding tokens, which have a normal scale, and returns the signal to a working range. The same happens if BOS sits at the start of the sequence.

In gated recurrent architectures, the read from the historical state is multiplied by a function of the current token — a gate or a linear query. With a zero input, the multiplier also goes to zero, and the contribution of the history shrinks. The residual vector stays near zero for several blocks in a row, and Pre-RMSNorm in every layer multiplies the backward gradients by a large factor. The effect accumulates with depth: input embedding gradients grow by more than 30 orders of magnitude (up to 35). Hybrids hit a ceiling; pure gated models keep growing.

The authors tested causality with an intervention: they opened the first-layer gate in the forward pass while keeping all backward derivatives, and the gradient dropped sharply. Disabling only the gate's backward derivative did not remove the problem.

Silent batch zeroing and two ways to fix it

Next, global-norm clipping kicks in. PyTorch, HuggingFace Trainer and FSDP2 sum the squares of the gradients in FP32. If even one entry is too large, squaring it yields infinity. The gradients are divided by the infinite norm and become zeros. AdamW then steps only on the old momentum and on weight decay for the rest of the network, while the new near-zero row, with zero momentum, stays frozen and breaks the following batches. This failure does not occur in inference or in on-policy RL rollouts, but a full-parameter RL update on prompts with near-zero chat tags triggers it.

The authors propose two solutions with no overhead in step time or peak memory:

• Separate clipping of input embeddings. The norm is computed separately from the rest of the weights, in 64-bit arithmetic or with rescaling, and it takes three lines of code. Update losses disappear, and the affected row "heals" in one step. There is no need to scan the vocabulary or change the architecture.
• A positive gate bias for pretraining from scratch. The gate stays positive at zero input, and the first layer passes a nonzero state. Near-zero gradients drop by more than 28 orders of magnitude on models up to 370M parameters, and validation perplexity is comparable to the baseline.

Disabling weight decay for embeddings will not help in checkpoints that have already been released: the explosion happens on the first step. And when training from scratch, disabling it alone is also not enough — the blocks still amplify gradients when norms are initialized small.

The practical takeaway is simple. If you fine-tune an untied gated checkpoint on a low-resource language or with your own chat tags, check the norms of the input embedding rows and move their clipping into a separate step.