TACO: a method removes credit “contamination” in RL training of language models

Researchers published on arXiv on July 10, 2026 the TACO method — Tail-Aware Credit Calibration — to improve reinforcement learning training of language models. The method eliminates a systemic flaw in RL algorithms: a situation where unlikely and contextually erroneous tokens receive the same positive credit as correct ones, gradually degrading the quality of reasoning.

Why uniform credit destroys reasoning

Critic-free RL methods for LLMs — such as GRPO — work by one principle: if a model's answer is recognized as correct, all tokens of that answer receive identical positive credit. The logic is clear: why differentiate tokens if the answer as a whole is correct?

The problem lies in the details. Among the tokens of a correct answer, inevitably there are "tail" tokens — unlikely, atypical for the given context. They may be syntactically or semantically erroneous locally, yet ended up in the correct answer "by chance." Receiving the same credit as correct tokens, they gradually teach the model to reproduce similar patterns.

The authors call this Positive-Credit Contamination — contamination by positive credit. The effect accumulates: the longer the RL session, the more erroneous patterns become embedded in the model.

• The problem affects all critic-free RL methods, including GRPO and REINFORCE
• "Tail" tokens — unlikely and atypical for the context — are reinforced equally with correct ones
• In long training sessions, standard methods often degrade in answer quality

How TACO calculates risk for each token

TACO adds one new component to the RL pipeline — a tail-risk score: an assessment for each token of how much it is an undesirable "tail" case.

The key idea is separating two types of rarity. First: "rare due to uncertainty" — the model is at a point of multiple equally acceptable options and exploring the answer space. This is normal, and such tokens continue to receive full credit. Second: "rare due to error" — a contextual anomaly that does not fit the distribution of correct answers. This is the type that is suppressed.

Meanwhile, TACO does not zero out the gradient of risky tokens completely, but only reduces their positive credit. This approach preserves useful rare patterns that occur systematically and gradually dampens random noise.

What the tests on eight benchmarks showed

The method was tested on three language models and eight standard reasoning evaluation benchmarks. TACO consistently outperformed baseline GRPO-based solutions across all datasets.

Most indicative is the result on training stability. Standard critic-free methods often degrade in long RL sessions: the credit signal becomes contaminated, quality stops growing. TACO provided steady growth in metrics throughout training without reversals — which is particularly important for long-term RL, which underlies modern reasoning models.

Source code published on GitHub: github.com/xiuyilou/TACO.

What this means

The problem of uniform credit in RL for LLMs has long been discussed in the community, but so far has not had a simple plug-in solution. TACO is compact and integrates on top of an existing pipeline without architecture overhaul. In 2025–2026, virtually all frontier labs use long-term RL as a key method for improving model reasoning capabilities. If TACO reproduces in larger experiments, the method could become a standard component of next-generation RL training.