Feedback Manipulation Regularization: Enabling Offline Agent Alignment for Imitation Learning
06:00 · July 11, 2026 · arXiv cs.AI RSS
Reinforcement learning (RL) research has increasingly shifted focus towards alignment, ensuring agents learn behaviors adhering to human values. While human demonstrations and feedback have proven crucial for alignment, existing approaches predominantly combine these signals using multi-stage pipelines designed for the contextual bandit framing of language generation. Yet little work explores how these complementary inputs can serve as a richer, interconnected signal for single-stage offline training in fully sequential decision-making environments. We propose Feedback Manipulation Regularization (FMR), an algorithm-agnostic method that harnesses evaluative feedback as a corrective signal to improve the alignment of imitation learning policies. We adapt Safety Gymnasium environments to be a principled testbed for alignment evaluation, demonstrating improved aptitude and up to a 98\% reduction in misalignment across a range of imitation learning algorithms. FMR remains robust in limited data regimes, even when learning from scarce aligned and uninformative noisy demonstrations.
Summary
Reinforcement learning research has increasingly emphasized alignment, the process of ensuring that agents adopt behaviors consistent with human values. Human demonstrations and evaluative feedback have both proven effective for this purpose, yet most existing techniques integrate the two signals through multi-stage pipelines originally developed for the contextual-bandit setting common in language-model training. Little attention has been given to methods that treat these signals as a single, interconnected source of supervision for fully sequential decision-making tasks trained entirely offline.
Feedback Manipulation Regularization (FMR) addresses this gap. The algorithm-agnostic technique uses evaluative feedback as a corrective regularizer that reshapes the imitation-learning objective during a single training stage. By penalizing actions that deviate from the feedback signal, FMR steers policies toward better-aligned behavior without requiring additional online interaction or separate fine-tuning phases.
The approach was evaluated on modified Safety Gymnasium environments configured as a controlled testbed for alignment. Across several imitation-learning algorithms, FMR produced measurable gains in task aptitude while reducing misalignment by as much as 98 percent. The same performance improvements held when training data were scarce or contained uninformative and noisy demonstrations, indicating that the regularization remains effective under realistic data constraints.
Why it matters
This research is highly relevant to the Dutch AI market's strong emphasis on ethical, transparent, and safe AI. FMR provides researchers with a robust method to align autonomous agents with human values, directly supporting the development of systems that comply with stringent EU AI safety standards.
