S2T-RLHF: почему в RLHF награда по предложениям стабильнее, чем по токенам

Why RLHF is unstable

Standard RLHF is unstable because the model receives a single scalar reward for the entire response, and this score is then "smeared" across policy updates at the level of individual tokens. Because of this, it's unclear exactly which part of the response earned the high or low score — the task of credit assignment within a response remains ambiguous.

A preference model, trained on pairwise comparisons of responses, evaluates a response only as a whole, not word by word. The preference signal is defined only at the response level, so mechanically transferring it to individual tokens introduces extra noise and destabilizes training.

What S2T-RLHF proposes

S2T-RLHF (sentence-to-token) introduces an intermediate level between the whole response and the individual token — the sentence. The method first decomposes the reward for the entire response across sentences, and within each sentence makes only a bounded adjustment at the token level. Importantly, this requires neither retraining the reward model nor preparing separate token-level reward labels.

• Three levels of granularity: whole response → sentence → token
• The reward is first split across sentences, then refined at the token level within bounded limits
• Does not require retraining the reward model
• Does not require token-level supervision — separate token-level reward labels
• Tested across several datasets and different optimization setups

According to the authors, this approach makes training more stable and robust, while preference alignment quality remains competitive — meaning stability doesn't come at the cost of accuracy.

Why sentence, not token

The sentence was chosen as the intermediate scale because it preserves semantic integrity while being more robust to noise than an individual token. The authors directly challenge the common assumption that finer-grained reward decomposition always leads to better optimization.

"When preference signals are noisy and defined only at the response

level, excessively fine-grained reward refinement can amplify uncertainty and destabilize training," the arXiv paper states.

Instead of chasing maximally fine-grained reward, the authors formulate a granularity-aware principle: reward design should be oriented toward stability, not maximal precision of distribution. The sentence level balances semantic coherence with robustness to token-level noise.

What this means

RLHF remains the primary way to align large language models with human preferences, but its instability is a persistent headache for engineers. S2T-RLHF offers an inexpensive recipe: instead of chasing maximally fine-grained reward, choose a sensible intermediate scale — the sentence. If the results are reproduced on other tasks and models, the "sentence level" could become the default granularity in preference training.

Frequently asked questions

What is RLHF?

RLHF (reinforcement learning from human feedback) is reinforcement learning based on human feedback. The model is fine-tuned to match human preferences: a reward model, trained on comparisons of responses, produces a reward that is used to adjust the generation policy.

Does S2T-RLHF require additional labeling?

No. According to the method's description, S2T-RLHF works without retraining the reward model and without token-level supervision. This means neither new training labels for token-level rewards nor separate evaluator training are needed — the method reuses the existing preference signal.