Structural entropy guided hierarchical symmetric multi-agent reinforcement learning

AbstractThe identification and analysis of physical priors in nature can significantly enhance the sample efficiency and performance of artificial intelligence models. In particular, leveraging symmetry priors in Multi-Agent Reinforcement Learning (MARL) effectively reduces the solution space, thereby improving algorithmic performance. While symmetry priors have been explored within MARL, existing methods often overlook the inherent hierarchical structures within Multi-Agent Systems (MAS). This paper introduces the concept of hierarchical symmetry, meaning that a MAS preserves symmetry across different levels, and bridges the existing gap by embedding this hierarchical symmetry into MARL. We propose the Hierarchical Symmetry-Informed Network (HSIN). HSIN incorporates the Equivariant Matrix Graph Network (EMGN), a novel network structure designed to process equivariant matrices composed of multiple equivariant variables of the agents. In addition, we employ structural entropy minimization to uncover the hierarchical structure of the MAS, identifying the optimal partitioning without the need for manually specified quotas. We provide a theoretical analysis demonstrating that HSIN strictly preserves symmetry, and we clarify the rationale behind the entropy-based partition method. Extensive experiments are conducted across diverse multi-agent environments, including the kinematic environment and SMACv2. The results show that compared to baseline algorithms, HSIN achieves faster convergence speed and higher convergence rewards. Additionally, we deploy HSIN in a real-world multi-robot environment, validating its effectiveness in practical applications.