CMU Researchers Teach RL Models to Stop Padding Answers: FORK Lifts Win Rate from 22% to 74%

Reinforcement learning on rubrics has become one of the working methods for improving models in areas where there's no automatic correctness check: medical consultations, long explanations, open-ended questions. But the method turned out to have a side effect familiar to anyone who's read answers from modern chatbots: they bloat. A group of researchers from Carnegie Mellon University and their colleagues showed that the cause lies directly in how the reward is structured, and proposed a fix called FORK. On medical benchmarks, it raised the win rate against the base model from 10.6% to 68.0% on RubricHub-Medical and from 22.1% to 74.1% on HealthBench.

Why a rubric rewards verbosity

A rubric is a list of criteria a good answer should satisfy: mention a contraindication, name a symptom, suggest seeing a doctor. The reward is usually calculated as a weighted share of satisfied criteria. In essence, this is recall: how much of what's needed the model covered.

The problem is that such a metric has no upper bound on length. Every extra phrase is one more chance to accidentally satisfy a criterion, and there's no penalty for junk. The RL policy quickly discovers this. Answers become more complete, but also more verbose, picking up unnecessary chunks and factual errors along the way. As a result, the trained model loses to the original one in direct comparison: people and judges prefer the base version's answers more, even though they're formally weaker on the rubric. A classic example of reward hacking, just without explicit cheating: the model is honestly optimizing exactly what's being measured.

What FORK adds

The authors introduce a second objective — precision. The requirement is simple: every unit of the answer must be both necessary and factually correct. Two mechanisms are used for this:

• necessity credits — credits derived from the rubric itself that show which fragment of the answer is actually needed to cover the criteria;
• a factual-correctness grader — a separate evaluator that checks the factual accuracy of each fragment.

FORK optimizes recall and precision jointly. An extra phrase that doesn't serve any criterion stops being free, and an incorrect statement directly lowers the reward. It becomes advantageous for the model to write tightly.

What the experiments showed

According to the paper (arXiv 2610.33512), FORK outperforms existing rubric-based RL methods and reaches a Pareto-optimal balance between rubric coverage and overall answer quality. Key results:

• compared to the original recall-based reward, precision is markedly higher, and response length is reduced;
• rubric coverage remains at a comparable level: the model doesn't sacrifice completeness for brevity;
• win rate against the base policy on RubricHub-Medical rose from 10.6% to 68.0%;
• win rate against the base policy on HealthBench rose from 22.1% to 74.1%.

The fact that base models were winning against RL versions in 78–89% of cases on its own is a telling figure. It shows that optimizing on the rubric without controlling precision produced formal gains on the metric and real degradation in the eyes of the evaluator.

The work was done by Hyun Ryu, Jong-Heon Kim, Yiqing Xie, Seungone Kim, Carolyn Rose, and Eunho Yang. The paper made it into Hugging Face Daily Papers, and the code is published in the JHKim631/FORK repository on GitHub.

What practitioners should do with this

Teams fine-tuning models on rubrics — especially in medicine, law, and finance, where the cost of an inaccurate extra statement is high — get a concrete checklist. First, look not just at rubric coverage but also at response length and the share of incorrect statements: growing length with a stable metric is a warning sign. Second, add a penalty for unnecessary content to the reward and a separate fact-check, rather than hoping the judge will punish the padding on its own. Third, compare the trained model against the base one in pairwise preferences: that's exactly where the reward hacking surfaced.

The limitation is obvious: FORK requires an additional factual-accuracy grader, which means additional computation at every training step and dependence on the quality of that grader. But the idea generalizes beyond medicine: anywhere an answer is evaluated against a list of criteria, a reward based on recall alone will push the model toward verbosity.