Brandon Reagen

ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Models

2024 · arXiv (Cornell University)

LayerNorm is a critical component in modern large language models (LLMs) for stabilizing training and ensuring smooth optimization. However, it introduces significant challenges in mechanistic interpretability, outlier feature suppression, faithful signal propagation, and computational and …

AERO: Softmax-Only LLMs for Efficient Private Inference

2024 · Qeios

The pervasiveness of proprietary language models has raised privacy concerns for users’ sensitive data, emphasizing the need for private inference (PI), where inference is performed directly on encrypted inputs. However, current PI methods face prohibitively …