CMU Researchers Teach RL Models to Stop Padding: FORK Lifts Win Rate from 22% to 74%

Reinforcement learning on rubrics has become one of the practical ways to improve models where correctness can't be checked automatically: medical consultations, long explanations, open-ended questions. But the method has a side effect familiar to anyone who reads modern chatbot answers: they bloat. A group of researchers from Carnegie Mellon University and their colleagues showed that the cause lies in the design of the reward itself, and proposed a fix called FORK. On medical benchmarks, it raised the win rate against the base model from 10.6% to 68.0% on RubricHub-Medical and from 22.1% to 74.1% on HealthBench.

Why a rubric rewards verbosity

A rubric is a list of criteria a good answer must satisfy: mention a contraindication, name a symptom, suggest seeing a doctor. The reward is usually computed as the weighted share of items satisfied. In essence, this is recall: how much of the required content the model covered.

The problem is that such a metric has no upper bound on length. Every extra sentence is another chance to hit a criterion by accident, and there is no penalty for junk. The RL policy finds this quickly. Answers become more complete, but also wordier, and they pick up unnecessary passages and factual errors. As a result, the trained model loses to the original in head-to-head comparison: people and judges prefer the base version's answers, even though by the rubric they are formally weaker. It is a classic case of reward hacking, only without any overt cheating: the model is honestly optimizing what it is measured on.

What FORK adds

The authors introduce a second objective, precision. The requirement is simple: every unit of the answer must be both necessary and factually correct. Two mechanisms do this:

• necessity credits, derived from the rubric itself, show which fragment of the answer is actually needed to cover the criteria;
• a factual-correctness grader, a separate evaluator, checks the factual accuracy of each fragment.

FORK optimizes recall and precision jointly. An extra sentence that serves no criterion is no longer free, and a false claim directly lowers the reward. The model is now better off writing densely.

What the experiments showed

According to the paper (arXiv 2610.33512), FORK outperforms existing rubric-based RL methods and reaches a Pareto-optimal balance between rubric coverage and overall answer quality. Key results:

• compared with the original recall-based reward, precision is noticeably higher and answer length goes down;
• rubric coverage stays at a comparable level: the model does not sacrifice completeness for brevity;
• win rate against the base policy on RubricHub-Medical rose from 10.6% to 68.0%;
• win rate against the base policy on HealthBench rose from 22.1% to 74.1%.

The fact that base models beat the RL versions 78–89% of the time is telling in itself. It means that optimizing against a rubric without controlling precision produced a formal gain on the metric and a real degradation in the evaluator's eyes.

The work was done by Hyun Ryu, Jong-Heon Kim, Yiqing Xie, Seungone Kim, Carolyn Rose and Eunho Yang. The paper made it onto Hugging Face Daily Papers, and the code is published in the JHKim631/FORK repository on GitHub.

What practitioners should take from this

Teams that fine-tune models on rubrics, especially in medicine, law and finance, where an inaccurate extra claim is costly, get a concrete checklist. First, look beyond rubric coverage to answer length and the share of incorrect claims: growing length with a stable metric is a warning sign. Second, add a penalty for unnecessary content and a separate fact check to the reward, rather than hoping the judge will "punish" the filler on its own. Third, compare the trained model with the base model in pairwise preferences: that is where the reward hacking surfaced.

The limitation is obvious: FORK requires an additional factual-accuracy grader, which means extra compute at every training step and a dependence on the quality of that grader. On the other hand, the idea carries over beyond medicine: wherever an answer is scored against a list of criteria, a recall-only reward will push the model toward verbosity.