Multi‐Agent Reinforcement Learning for Multi‐Robot Warehouse Automation in a 3D Environment Using Unity Game Engine

ABSTRACT Multiagent reinforcement learning (MARL) has emerged as a key enabling technology for intelligent coordination in cyber‐physical robotic systems, such as automated warehouses. However, the effectiveness of MARL in real‐world scenarios is often hindered by challenges, such as sparse reward signals and partial observability, which limit the agents' ability to learn optimal behaviours. In this paper, we propose a novel method to design MARL for warehouse management and implement it within a game engine to facilitate real‐world visualisation for industry users. The contributions of this paper are twofold. First, we introduce temporal attention‐enhanced counterfactual multiagent reinforcement learning (TA‐COMA), a novel method designed to enhance performance in environments with sparse rewards and partial observations by integrating temporal advantages. Second, we propose a new approach to define intermediate rewards in sparse reward environments, exemplified through a warehouse scenario. We also demonstrate, for the first time, the implementation of a multiagent reinforcement learning strategy for warehouse environments within the Unity game engine, utilising the ML‐Agents toolkit. The simulation results demonstrate that intermittent rewards significantly improve learning performance. TA‐COMA consistently achieves higher asymptotic mean cumulative rewards than COMA, PPO and shared experience actor–critic (SEAC). We further show that the variance of the temporal advantage in TA‐COMA is reduced by approximately compared to COMA, improving temporal coherence whilst preserving informative advantage structure and stabilising learning. The Unity platform further enables the creation of a realistic 3D warehouse environment and immersive visualisation of multirobot behaviour using a virtual reality (VR) headset.