LALR: Theoretical and Experimental validation of Lipschitz Adaptive\n Learning Rate in Regression and Neural Networks

We propose a theoretical framework for an adaptive learning rate policy for\nthe Mean Absolute Error loss function and Quantile loss function and evaluate\nits effectiveness for regression tasks. The framework is based on the theory of\nLipschitz continuity, specifically utilizing the relationship between learning\nrate and Lipschitz constant of the loss function. Based on experimentation, we\nhave found that the adaptive learning rate policy enables up to 20x faster\nconvergence compared to a constant learning rate policy.\n