S3AC Brings the Critic Back to RL for LLMs: Gains from 1.5B to 397B Parameters

Researchers at Shanghai AI Laboratory have proposed S3AC (Stabilized, Selective, and Scalable Actor-Critic), a practical recipe for training a value critic for RL with verifiable rewards on long reasoning chains. The method was tested on dense and MoE models from 1.5B to 397B parameters. It outperformed GRPO-X, a modern GRPO variant. At 35B, S3AC kept training with a single rollout per prompt, while critic-free baselines began to degrade after their first improvements.

Why the industry ran away from the critic

In RLVR for long CoT, the critic has long been considered an expensive and temperamental component. It has to be trained separately, it eats memory and compute, and it behaves unstably on long trajectories. As a result, most teams use critic-free methods such as GRPO: they generate a group of responses for each prompt and compute the baseline from the group's mean result.

This approach has two known weaknesses. First, it lacks token-level credit assignment. The whole trajectory gets a single score, even though the error may have occurred in a few specific reasoning steps. Second, it has to generate groups of rollouts just to obtain a baseline for a prompt. That is expensive, and on tasks that are too easy or too hard, some of the generations are wasted.

A critic solves both problems: it produces per-token advantages and a baseline for each prompt. The S3AC authors explain why it never took off on long CoT. First, trajectories of different lengths are poorly represented in the critic's training data. Second, early in training a freshly initialized value head dominates the gradients and destabilizes the rest of the network.

The S3AC recipe and prompt selection before generation

The main contribution of the work is a set of techniques that make critic training stable on long reasoning. The trained critic provides both signals that are needed: token-specific advantages and prompt-specific baselines.

The second idea is Difficulty-Aware Prompt Sampling (DAPS). The critic's baseline shows how hard a prompt is for the model right now. DAPS combines this estimate with the prompt's outcome history and chooses which tasks are worth sending to generation. Selection happens before rollouts, so no generations are wasted. No offline difficulty initialization is needed: there is no need to run the dataset through the model in advance and label it by difficulty.

• Method: S3AC, an actor-critic for LLM reasoning with verifiable rewards
• What the critic provides: token-specific advantages and prompt-specific baselines
• Data selection: DAPS, which picks prompts before generation based on a difficulty estimate and outcome history
• Model range: dense and mixture-of-experts, from 1.5B to 397B parameters
• Comparison baseline: GRPO-X, a modern GRPO variant
• Single-rollout mode: at 35B, S3AC keeps learning, while the critic-free methods tested degrade after their first improvements
• Team: Shuo Shen, Guangran Cheng, Chengqi Lyu, Xiangsen Wang, Yanan Sun, Wenwei Zhang
• Code: InternLM/xtuner repository on GitHub
• Publication: arXiv 2610.35518, Hugging Face Daily Papers

The most telling result is the single-rollout mode. Without a group, group-based methods lose their baseline, and at 35B their training sags after the initial gain. The critic estimates the baseline itself, so quality does not drop. And since the method scales to 397B, this is not a lab toy.

Who should look at S3AC, and when

Teams that fine-tune reasoning models with RLVR run into the cost of rollouts. Selecting prompts by difficulty before generation saves exactly that line item. If a group of responses to a task is bound to produce the same result, it carries almost no training signal. DAPS tries not to spend compute on such tasks.

A second practical advantage is stability with a small number of rollouts per prompt. This is useful where generating long chains is very costly, for example on large MoE models. The implementation is available in xtuner, so you can test the method on your own tasks without writing a pipeline from scratch.

The limitations are also clear: the critic remains an additional component, and the advantage over GRPO-X has been shown on the authors' own set of models and configurations. Independent reproductions will show how well the recipe transfers to other reward domains and other model families.