Naman Agarwal

Acceleration via Fractal Learning Rate Schedules

2021 · arXiv (Cornell University)

In practical applications of iterative first-order optimization, the learning rate schedule remains notoriously difficult to understand and expensive to tune. We demonstrate the presence of these subtleties even in the innocuous case when the objective …

Training neural networks faster with minimal tuning using pre-computed lists of hyperparameters for NAdamW

2025 · arXiv (Cornell University)

If we want to train a neural network using any of the most popular optimization algorithms, we are immediately faced with a dilemma: how to set the various optimization and regularization hyperparameters? When computational resources …