Ebert, Jan

Optimal Scaling Needs Optimal Norm

2025 · arXiv (Cornell University)

Despite recent progress in optimal hyperparameter transfer under model and dataset scaling, no unifying explanatory principle has been established. For Adam and Scion optimizers, we discover that joint optimal scaling across model and dataset sizes …