E Weinan

On the emergence of tetrahedral symmetry in the final and penultimate layers of neural network classifiers.

2020 · arXiv (Cornell University)

A recent numerical study observed that neural network classifiers enjoy a large degree of symmetry in the penultimate layer. Namely, if $h(x) = Af(x) +b$ where $A$ is a linear map and $f$ is the …

On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers

2020 · arXiv (Cornell University)

A recent numerical study observed that neural network classifiers enjoy a large degree of symmetry in the penultimate layer. Namely, if $h(x) = Af(x) +b$ where $A$ is a linear map and $f$ is the …

GradPower: Powering Gradients for Faster Language Model Pre-Training

2025 · ArXiv.org

We propose GradPower, a lightweight gradient-transformation technique for accelerating language model pre-training. Given a gradient vector $g=(g_i)_i$, GradPower first applies the elementwise sign-power transformation: $φ_p(g)=({\rm sign}(g_i)|g_i|^p)_{i}$ for a fixed $p>0$, and then feeds the transformed …