FMR method cuts RL agent misalignment by 98% in imitation learning

Researchers published an arXiv preprint in July 2026 describing Feedback Manipulation Regularization (FMR), a single-pass offline method for aligning agents in imitation learning that reduces the rate of value-constraint violations by 98%.

Why a new alignment method is needed

Most current approaches to aligning RL agents, meaning training systems to act in line with human values, rely on multi-stage pipelines. The agent first learns from demonstrations, then receives evaluative feedback from humans, then goes through a separate fine-tuning stage, as in RLHF.

These schemes were originally designed for language models in a "contextual bandit" format, where the whole sequence of actions collapses into generating a single response. For fully sequential environments such as robotics, game-playing agents and autonomous systems, this architecture works poorly. The agent makes decisions step by step, and even a small deviation from value constraints accumulates along the whole chain of actions.

How FMR works

FMR tackles this problem by building evaluative feedback directly into the imitation learning process as a regularizer, with no separate RLHF stage or multi-stage pipeline.

Key properties of the method:

• Algorithmic neutrality: FMR works on top of any existing imitation learning algorithm and does not require changes to the base architecture
• Single-pass offline training: the whole alignment process is one stage, with no switching between pipelines
• Robustness to data scarcity: the method stays effective with sparse and noisy training demonstrations
• A corrective signal, not an extra reward: the feedback shifts the training distribution toward aligned behavior rather than adding yet another reward function

What the Safety Gymnasium tests showed

To test the method, the authors adapted Safety Gymnasium, a set of simulated environments designed specifically to evaluate safe agent behavior. FMR was tested on top of several baseline imitation learning algorithms.

The results were notable: across the full range of algorithms tested, the method reduced the rate of value-constraint violations by up to 98%. That means the agent breaks the given human constraints far less often, without losing quality in imitating the target behavior.

"FMR remains robust under limited data, even when trained on sparse

aligned and information-poor noisy demonstrations," the authors say in the preprint.

Importantly, the improvement showed up in two competing metrics at once: the agent copied demonstrations more accurately and violated value constraints less often.

What this means

FMR is an algorithm-agnostic, practical way to improve the alignment of RL agents without rebuilding the whole training setup. For developers working on real sequential tasks such as robotics, autonomous systems and simulation, it is potentially a ready-to-use component that slots into an existing stack. If the results hold up on broader benchmarks, FMR could become part of the standard toolkit wherever alignment errors have real consequences.