Differential graph game-based hierarchical cooperative reinforcement learning control for in-orbit assembly with variable topology

<p indent="0mm">To address the challenge of collaborative control in the interconnection of multi-subsystem assemblies operating within complex environments, this study introduces a hierarchical framework grounded in reinforcement learning, leveraging zero-sum games and differential graphical games. The inner-layer control employs zero-sum differential games to devise distributed robust strategies, effectively mitigating issues arising from subsystem coupling, external disturbances, and inherent uncertainties. At the system level, the outer-layer control employs differential graphical games to coordinate subsystem behaviors, ensuring global optimality and achieving Nash equilibrium. The collaborative stability of the inner-layer control is rigorously established via the loop small-gain theorem, while the interplay between local and global optimality is critically analyzed. The validity of the outer-layer strategy is further corroborated through Nash equilibrium. Expanding upon this, a distributed multi-agent reinforcement learning approach is developed, wherein an advantage function is meticulously designed to minimize the output of the evaluation network. This innovation enables autonomous adjustment of action networks by individual subsystems, facilitating collaborative optimization while addressing challenges such as excessive variance and sluggish convergence in control policies. Simulation results underscore the efficacy and adaptability of the proposed method, highlighting its potential for complex on-orbit assembly applications.