Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Control
06:00 · June 24, 2026 · arXiv cs.AI RSS
Multi-agent systems are widely used in safety-critical applications that require coordinated behavior under strict safety constraints. Existing approaches face a fundamental trade-off: learning-based methods achieve strong empirical performance but lack theoretical safety guarantees, while control-theoretic methods enforce safety but often lead to overly conservative and inefficient behaviors. We propose a hierarchical multi-agent reinforcement learning framework that enforces hard safety constraints under mild assumptions at low level via a constraint manifold, while enabling effective coordination through high-level policy learning. Our approach provides theoretical safety guarantees in the multi-agent setting and yields stationary learning dynamics, thereby enabling stable and efficient training. Empirically, our method achieves competitive performance while maintaining nearly perfect safety rates, and generalizes effectively to varying numbers of agents and obstacles.
Summary
Multi-agent systems deployed in domains such as warehouse robotics and drone swarms must coordinate tasks while satisfying strict safety constraints, notably collision avoidance. Existing learning-based methods deliver strong task performance yet provide no formal safety assurances, whereas purely control-theoretic techniques enforce hard constraints but frequently produce conservative behavior and deadlocks in cluttered environments.
The proposed hierarchical framework addresses this trade-off by separating concerns across two layers. A high-level policy, trained under centralized training with decentralized execution, generates subgoals that manage inter-agent coordination and long-horizon planning. At the low level, a model-based controller projects actions onto the tangent space of a constraint manifold, thereby embedding hard safety requirements directly into the feasible action set without repeated quadratic-program solves.
Under mild assumptions the manifold construction yields formal safety guarantees that hold at every environmental timestep for both training and execution, while the resulting learning dynamics remain stationary. This combination supports stable policy optimization and avoids the hyperparameter sensitivity typical of Lagrangian or control-barrier-function approaches.
Empirical evaluation on lidar-based navigation benchmarks shows that the method matches or exceeds the task performance of prior safe multi-agent reinforcement learning baselines while maintaining near-perfect safety rates. Policies trained on small instances further generalize to substantially larger teams and obstacle counts without retraining, preserving both safety and success metrics.
Why it matters
This research is highly relevant for Dutch AI researchers focusing on autonomous systems, robotics, and logistics, where safety-critical multi-agent coordination is essential. It aligns with the Netherlands' strategic emphasis on safe, reliable, and transparent AI by providing theoretical safety guarantees in reinforcement learning.
