Leonardo Cofone

Noise Accumulation and Rank Collapse in Dense Self-Attention: DSALT

2026 · Zenodo (CERN European Organization for Nuclear Research)

Large language models based on the Transformer decoder architecture perform multi-head self-attention over all previous tokens in the context window. We argue that this dense attention mechanism introduces a systematic form of noise: every token, …