DLC Correction Keeps KL to On-Policy Training Within Two Noise Floors on Qwen3

In RL training of language models, one inference engine, such as vLLM, generates the responses, while a separate trainer computes the gradients. Formally, the training counts as on-policy: responses should come from the current policy. In practice, the two systems assign different probabilities to tokens, and the data becomes off-policy. A new paper on arXiv explains where this mismatch comes from and how it develops during training. The authors propose a correction called DLC (discrepancy-lag correction). With a stale sampler, it keeps the KL divergence to on-policy training within two noise floors. Here, a noise floor is the KL between two independent on-policy training runs. The result was obtained on Qwen3-0.6B and Qwen3-1.7B.

Two sources of mismatch

The first source is the engine's numerical imprecision. It has its own kernels and computation precision, so even on the same weights it produces a distribution that differs from the trainer's. Quantized samplers widen this gap. The second source is stale weights in asynchronous generation. The sampler holds the weights from the last sync while the trainer moves ahead. Asynchronous systems such as AReaL work this way.

The authors run the trainer on the sampler's weights and obtain an exact decomposition of the per-token error into two terms: numerical discrepancy and policy lag. The numerical discrepancy stays roughly constant, so it shifts the updates by a nearly fixed amount. Centering the score at each prefix removes most of it.

Policy lag is harder. The trainer's own updates produce it. The bias left over after correction feeds into the next update, which increases the next lag, and a feedback loop forms.

Importance weighting breaks the loop only while the probability ratio stays concentrated. The boundary is set by the effective sample size (ESS) of that ratio: when it falls below 0.5, the method stops working.

On Qwen3-1.7B, because of the heavy tail of the ratio, ESS crosses the threshold at step 11, three steps earlier than the Gaussian prediction. This is exactly the step at which importance sampling leaves the noise floor.

In addition, the excess update norm of score centering relative to importance sampling contains a positive quadratic term in the product of learning rate and number of steps.

How DLC works

The correction mirrors this decomposition and has three parts:

• it centers the numerical discrepancy;
• it clips the lag ratio relative to the sampler's weights;
• it resyncs the sampler when the ESS of the lag ratio falls below 0.5.

With a stale sampler, DLC keeps Qwen3-1.7B at the noise floor through step 40. In three DLC runs, the sync triggered on the condition of ESS below 0.5. With importance sampling, ESS stays below 0.5 after step 11. On both models, the deviation from on-policy training does not exceed two noise floors. Forced synchronization prevents the collapses that remain with per-token corrections. Ordinary KL measured on the sampler's own prefixes does not detect these collapses.

Why earlier corrections behave differently

The work explains the spread of results in earlier studies. Truncated importance sampling weights each token by a truncated probability ratio. Score centering subtracts the sampler's mean score at the prefix. PPO clips the ratio of the trainer to a reference policy. Centering works reliably with a quantized sampler but loses to importance sampling with a stale one: in earlier experiments, the sampler was synced once every 64 steps. PPO survives staleness but breaks under quantization and synthetic weight noise. The collapse of asynchronous GRPO under staleness grows with the product of staleness and learning rate.

The authors also separate out an infrastructure layer. Top-p sampling, re-tokenization of text, filtering responses by reward, and deferred long responses also create mismatch. These are removed at the infrastructure level: the sampler reports the distribution it actually used, the trainer receives token IDs, and batches are formed independently of outcome. After that, only the weight version and the engine numerics reach the algorithm. Weight lag is addressed by decoupled PPO, M2PO, and VCPO, which scales the learning rate by ESS. Numerical discrepancy is corrected by truncated importance sampling and score centering.

The practical takeaway: the choice of correction depends on whether the error grows with updates. There is no universal fix. In asynchronous RL, it is useful to monitor the ESS of the weight ratio and treat it as a signal to sync. The experiments are limited to Qwen3-0.6B and Qwen3-1.7B, so transfer to larger models needs to be verified separately.