HyPOLE: Hyperproperty-Guided Multi-Agent Reinforcement Learning under Partial Observation
06:00 · July 1, 2026 · arXiv cs.AI RSS
Formal specification is a powerful tool to guide the learning process and provides significant advantages over reward shaping: (1) mathematical rigor; (2) expressiveness to specify objectives and constraints, and (3) the ability to define tactics to achieve objectives. However, these benefits remain largely unexplored in the context of Multi-Agent Reinforcement Learning (MARL). This paper introduces HyPOLE, a novel framework for MARL under partial observability, where learning is guided by the expressive power of the so-called hyperproperties and, in particular, the temporal logic HyperLTL. We integrate Centralized Training for Decentralized Execution (CTDE) techniques with HyPOLE to synthesize decentralized policies, and our evaluation on SMAC, MessySMAC, and WildFire benchmark demonstrates clear advantages over baselines.
Summary
Formal specification offers a structured alternative to reward shaping in reinforcement learning by supplying mathematical rigor, the capacity to express both objectives and constraints, and explicit guidance on how those objectives can be achieved. These properties have received little attention in multi-agent reinforcement learning (MARL), especially when agents must act on incomplete local observations. HyPOLE addresses this gap by embedding hyperproperties expressed in HyperLTL into the training loop, allowing the learning process to be constrained by temporal relations that span multiple execution traces rather than single-agent rewards.
The framework combines the centralized-training decentralized-execution (CTDE) paradigm with HyperLTL monitors. During centralized training, a joint critic evaluates candidate joint policies against the hyperproperty specification; at execution time, each agent deploys an independent policy that has been shaped by the same specification. This separation preserves the scalability of decentralized control while retaining the formal guarantees available only when global information is accessible during learning.
Empirical evaluation on the SMAC, MessySMAC, and WildFire benchmarks indicates consistent improvements over standard MARL baselines that rely solely on reward shaping. The results suggest that the added expressiveness of hyperproperties can be integrated into existing CTDE pipelines without sacrificing sample efficiency or final policy quality.
Why it matters
The use of formal specifications to guide MARL aligns strongly with the Dutch and EU focus on robust, transparent, and verifiable AI systems. This research provides advanced methodologies for Dutch AI researchers developing safe multi-agent systems for complex, partially observable environments.
