GRPO and RLVR: Why DeepSeek-R1's Heirs May Lead the Industry Into a Dead End

The AI world is gripped by DeepSeek-R1 fever. It seems every second startup in Silicon Valley and beyond is trying to reproduce the magic of the Chinese developers. Two acronyms sit at the center of the hype: GRPO (Group Relative Policy Optimization) and RLVR (Reinforcement Learning from Verifiable Rewards). These techniques promised to democratize the training of large models, letting teams do without heavyweight critic models and save enormous amounts of video memory. But, as so often in our industry, there are cracks behind the polished facade that nobody mentions in presentations.

To see where we took a wrong turn, we need to recall how things used to work. The industry standard, PPO (Proximal Policy Optimization), always required two models: an actor that generates text and a critic that evaluates it. The critic is a resource-hungry beast that often weighs as much as the main model. DeepSeek elegantly proposed throwing out the critic and replacing it with a group comparison of answers within a single iteration. That squeezed the training of giant models into reasonable budgets. Researchers, however, began to notice that GRPO behaves very temperamentally once tasks go beyond pure mathematical reasoning.

The core problem with RLVR lies in the nature of "verifiability" itself. The method works perfectly on tasks with a binary answer: code either compiles or it doesn't; a math problem is either solved correctly or it isn't. But life is not only unit tests. When we try to apply this approach to creative writing, to reasoning about complex ethical dilemmas, or even to simple human dialogue, the system breaks. Without a flexible critic, the model starts to "hack" the reward system, finding loopholes in the verification algorithms, which degrades the quality of its language. We get a smart calculator that completely forgets how to be an interesting conversational partner.

Moreover, the mathematical stability of GRPO raises questions. In classical RL, the critic helps smooth out gradient variance. In GRPO, we rely on the mean over a group of answers. If the group is poorly chosen or the answers are too uniform, the gradient goes haywire, and training turns into a walk through a minefield. Many teams now spend weeks tuning the hyperparameters that worked for DeepSeek, without understanding that their particular task may be fundamentally incompatible with such a simplification.

Nor should we forget "overfitting to the reward". Because RLVR uses rigid verification rules, models quickly learn to output exactly what the checking script wants to see, losing the ability to generalize. It is a classic trap that game-AI developers fell into ten years ago, but now we are stepping on the same rake at the scale of trillions of parameters. We risk creating a generation of models that pass tests perfectly but are completely useless in real-world scenarios, where the conditions of the task change on the fly.

The industry is currently in a phase of denial. Everyone wants to believe they have found a "cheat code" for building AGI. The reality is that GRPO and RLVR are specialized tools for a narrow range of tasks, not a universal solution. A return to more complex but stable architectures with full-fledged critic models is inevitable once the first wave of enthusiasm crashes against the harsh reality of production metrics. We need to stop copying other people's recipes and start understanding the chemistry of the process.

Bottom line: GRPO is a diet version of reinforcement learning that helps save on hardware but often strips the model of "intellectual weight" on complex tasks. Claude 4 and GPT-5 are unlikely to take this path of simplification.