FMR Method Reduces RL Agent Alignment Errors by 98% in Imitation Learning

Researchers published a preprint on arXiv in July 2026 describing Feedback Manipulation Regularization (FMR) — a method of single-pass offline alignment of agents during learning with imitation, which reduces the frequency of violations of value constraints by 98%.

Why a new alignment method is needed

Modern approaches to RL agent alignment — that is, training systems to act in accordance with human values — are in most cases built on multi-stage pipelines. First, the agent learns from demonstrations, then receives evaluative feedback from humans, then goes through a separate fine-tuning stage — as in RLHF.

Such schemes were originally created for language models in the format of a "contextual bandit": the entire sequence of actions is reduced to generating a single response. For fully sequential environments — robotics, game agents, autonomous systems — this architecture works poorly. The agent makes decisions step by step, and even slight deviation from value constraints accumulates over the entire chain of actions.

How FMR works

FMR solves this problem by embedding evaluative feedback directly into the imitation learning process as a regularizer — without a separate RLHF stage or multi-stage pipeline.

Key properties of the method:

• Algorithmic neutrality: FMR works on top of any existing imitation learning algorithm without requiring changes to the base architecture
• Single-pass offline training: the entire alignment process is one stage, without switching between pipelines
• Robustness to data scarcity: the method maintains efficiency with scarce and noisy training demonstrations
• Corrective signal, not additional reward: feedback shifts the learning distribution toward aligned behavior rather than adding another reward function

What tests in Safety Gymnasium showed

To verify the method, the authors adapted the Safety Gymnasium platform — a set of simulated environments developed specifically to evaluate safe agent behavior. FMR was tested on top of several base imitation learning algorithms.

The results proved significant: across the full range of tested algorithms, the method reduced the level of non-compliance with value constraints to 98%. This means the agent significantly less often violates specified human constraints — and doesn't lose quality in imitating target behavior.

"FMR remains robust under limited data conditions, even when training

on scarce aligned and information-poor noisy demonstrations," the authors report in the preprint.

Importantly, the improvement manifested in two competing metrics: the agent became more accurate at copying demonstrations and less often violated value constraints.

What this means

FMR is an algorithmically neutral and practically applicable way to improve RL agent alignment without completely rebuilding the learning system. For developers working with real sequential tasks — robotics, autonomous systems, simulation — this is a potentially ready-to-use component that can be embedded in an existing stack. If results are confirmed on wider benchmarks, FMR could become part of the standard toolkit where alignment errors have real consequences.