Ruoyu Sun

On the Convergence of A Class of Adam-Type Algorithms for Non-Convex\n Optimization

2018 · arXiv (Cornell University)

This paper studies a class of adaptive gradient based momentum algorithms\nthat update the search directions and learning rates simultaneously using past\ngradients. This class, which we refer to as the "Adam-type", includes the\npopular algorithms such as …

Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learning

2022 · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Model-agnostic meta-learning (MAML) and its variants have become popular approaches for few-shot learning. However, due to the non-convexity of deep neural nets (DNNs) and the bi-level formulation of MAML, the theoretical properties of MAML with …

NTK-SAP: Improving neural network pruning by aligning training dynamics

2023 · arXiv (Cornell University)

Pruning neural networks before training has received increasing interest due to its potential to reduce training time and memory. One popular method is to prune the connections based on a certain metric, but it is …

Towards Quantifying the Hessian Structure of Neural Networks

2025 · arXiv (Cornell University)

Empirical studies reported that the Hessian matrix of neural networks (NNs) exhibits a near-block-diagonal structure, yet its theoretical foundation remains unclear. In this work, we reveal that the reported Hessian structure comes from a mixture …