Regularize! Don't Mix: Multi-Agent Reinforcement Learning without\n Explicit Centralized Structures

We propose using regularization for Multi-Agent Reinforcement Learning rather\nthan learning explicit cooperative structures called {\\em Multi-Agent\nRegularized Q-learning} (MARQ). Many MARL approaches leverage centralized\nstructures in order to exploit global state information or removing\ncommunication constraints when the agents act in a decentralized manner.\nInstead of learning redundant structures which is removed during agent\nexecution, we propose instead to leverage shared experiences of the agents to\nregularize the individual policies in order to promote structured exploration.\nWe examine several different approaches to how MARQ can either explicitly or\nimplicitly regularize our policies in a multi-agent setting. MARQ aims to\naddress these limitations in the MARL context through applying regularization\nconstraints which can correct bias in off-policy out-of-distribution agent\nexperiences and promote diverse exploration. Our algorithm is evaluated on\nseveral benchmark multi-agent environments and we show that MARQ consistently\noutperforms several baselines and state-of-the-art algorithms; learning in\nfewer steps and converging to higher returns.\n