TACO: A Method That Removes Credit "Contamination" in RL Training of Language Models

Researchers published TACO (Tail-Aware Credit Calibration) on arXiv on July 10, 2026, a method for improving reinforcement learning of language models. It fixes a systemic flaw in RL algorithms: low-probability, contextually wrong tokens receive the same positive credit as correct ones, gradually degrading reasoning quality.

Why uniform credit damages reasoning

Critic-free RL methods for LLMs, such as GRPO, work on one principle: if the model's answer is judged correct, every token in that answer gets the same positive credit. The logic is easy to follow: why differentiate between tokens if the answer as a whole is right?

The problem lies in the details. Any correct answer inevitably contains "tail" tokens: low-probability, atypical for the given context. They may be locally wrong, syntactically or semantically, yet ended up in a correct answer "by chance." Because they receive the same credit as correct tokens, they gradually teach the model to reproduce similar patterns.

The authors call this Positive-Credit Contamination. The effect accumulates: the longer the RL session, the more erroneous patterns become entrenched in the model.

• The problem affects all critic-free RL methods, including GRPO and REINFORCE
• "Tail" tokens, which are low-probability and atypical for the context, are reinforced just as much as correct ones
• In long training sessions, standard methods often degrade in answer quality

How TACO computes each token's risk

TACO adds one new component to the RL pipeline: a tail-risk score, an estimate for each token of how far it is an undesirable "tail" case.

The key idea is to separate two kinds of rarity. The first is "rare because of uncertainty": the model stands at a point with several equally valid options and is exploring the space of answers. This is normal, and such tokens keep receiving full credit. The second is "rare because of error": a contextual anomaly that does not fit the distribution of correct answers. This is the type TACO suppresses.

TACO does not zero out the gradient of risky tokens entirely; it only reduces their positive credit. This approach preserves useful rare patterns that appear systematically and gradually dampens random noise.

What the tests on eight benchmarks showed

The method was tested on three language models and eight standard reasoning benchmarks. TACO consistently outperformed GRPO-based baselines on every set.

The most telling result concerns training stability. Standard critic-free methods often degrade in long RL sessions: the credit signal gets polluted and quality stops improving. TACO delivered steady gains throughout training with no regressions, which matters especially for the long-horizon RL underlying modern reasoning models.

The source code is available on GitHub: github.com/xiuyilou/TACO.

What this means

The problem of uniform credit in LLM RL has long been discussed in the community, but until now lacked a simple drop-in fix. TACO is compact and plugs into an existing pipeline without architectural changes. In 2025–2026, nearly all frontier labs use long-horizon RL as a key method for improving model reasoning. If TACO reproduces in larger experiments, it may become a standard component of next-generation RL training.