Pavan Katta

On Implications of Scaling Laws on Feature Superposition

2024 · arXiv (Cornell University)

Using results from scaling laws, this theoretical note argues that the following two statements cannot be simultaneously true: 1. Superposition hypothesis where sparse features are linearly represented across a layer is a complete theory of …