Nora Belrose

Converting MLPs into Polynomials in Closed Form

2025 · arXiv (Cornell University)

Recent work has shown that purely quadratic functions can replace MLPs in transformers with no significant loss in performance, while enabling new methods of interpretability based on linear algebra. In this work, we theoretically derive …