A New Approach to Neural Network Optimization: PID-based Optimization

We introduce a Neural Network optimization model inspired by the renowned Proportional-Integral-Derivative (PID) controller. This method leverages the principles of the PID controller to effectively reduce loss errors during training. By applying the theory behind PID control, our approach aims to achieve higher accuracy across diverse objectives and problems involving large datasets. Additionally, it is well-suited for non-stationary objectives with noisy or sparse gradients. The hyperparameters of our PID-inspired optimizer have intuitive interpretations. The connections between these hyperparameters and the PID controller are thoroughly discussed. We tested the PID optimizer across various scenarios, demonstrating its superior performance and robustness.