Simulation Diversity: An Approach for Accurate Performance Estimation in Reinforcement Learning

In the realm of Reinforcement Learning research, the Sim-to-Real (S2R) gap poses a persistent challenge. This gap manifests as a decline in the agent’s performance in real-world scenarios in contrast to simulations. Addressing this issue is crucial for enhancing decision-making in the deployment of agents to reality. Therefore, we propose a novel technique, named Simulation Diversity (SD), which consists of employing multiple simulators to articulate similar dynamics, thereby enhancing the accuracy of performance estimation. Quantitative metrics, including Wasserstein Distance, Energy Distance, and Mean Squared Error provide evidence of the improved capabilities of SD in mimicking real-world agent performance. The proposed approach is further validated by way of micro-accuracy assessments, showcasing that SD is a much better predictor of the reward gained by the agent. The paper provides valuable insights into the rationale behind the increased accuracy achieved by SD. Experimental results demonstrate a comparative analysis between simulator-based estimations of agent performance and the actual performance of a deployed policy in the real world.