S2T-RLHF: Why Sentence-Level Rewards Make RLHF More Stable Than Token-Level Ones

In July 2026, a paper appeared on arXiv describing S2T-RLHF, a method for training language models on human preferences. It distributes reward first across the sentences of a response and then across individual tokens, which makes the notoriously unstable RLHF process noticeably more robust.

Why RLHF is unstable

Standard RLHF is unstable because the model receives a single scalar reward for the entire response, and that score is then "smeared" across policy updates at the level of individual tokens. As a result, it is unclear which part of the response actually earned a high or low score. The credit assignment problem within a response remains ambiguous.

A preference model trained on pairwise comparisons of responses scores only the response as a whole, not individual words. The preference signal exists only at the response level, so transferring it mechanically to individual tokens adds extra noise and destabilizes training.

What S2T-RLHF proposes

S2T-RLHF (sentence-to-token) introduces an intermediate level between the whole response and the individual token: the sentence. The method first breaks the response-level reward down across sentences, then makes only a bounded adjustment across tokens within each sentence. One important detail: it requires neither retraining the reward model nor preparing separate token-level reward annotations.

• Three levels of granularity: whole response → sentence → token
• The reward is first split across sentences, then refined across tokens in a bounded way
• No reward model retraining required
• No token-level supervision required, meaning no separate annotation of rewards per token
• Tested on several datasets and under different optimization settings

According to the authors, this approach makes training more stable and robust, while preference alignment quality stays competitive. In other words, stability does not come at the cost of accuracy.

Why a sentence and not a token

The sentence was chosen as the intermediate scale because it preserves semantic coherence while being more robust to noise than an individual token. The authors directly challenge the common assumption that the finer the reward is split, the better optimization goes.

"When preference signals are noisy and specified only at the response

level, overly fine-grained reward refinement can amplify uncertainty and destabilize training," the paper on arXiv states.

Instead of chasing the most granular reward possible, the authors formulate a granularity-aware principle: reward design should prioritize stability over maximum precision of distribution. The sentence level balances semantic coherence with robustness to token-level noise.

What this means

RLHF remains the main way to align large language models with human preferences, but its instability is a constant headache for engineers. S2T-RLHF offers a cheap recipe: don't chase the finest-grained reward, but pick a sensible intermediate scale, the sentence. If the results replicate on other tasks and models, "sentence level" could become the default granularity in preference training.

Frequently asked questions

What is RLHF?

RLHF (reinforcement learning from human feedback) is reinforcement learning on human feedback. The model is fine-tuned to match human preferences: a reward model trained on comparisons of responses produces a reward, and that reward is used to adjust the generation policy.

Does S2T-RLHF require additional annotation?

No. According to the method's description, S2T-RLHF works without retraining the reward model and without token-level supervision. That means it needs neither new training annotations of rewards per token nor separate training of the scorer; the method reuses the preference signal that already exists.