António V. Lopes

One Wide Feedforward is All You Need

2023 · arXiv (Cornell University)

The Transformer architecture has two main non-embedding components: Attention and the Feed Forward Network (FFN). Attention captures interdependencies between words regardless of their position, while the FFN non-linearly transforms each input token independently. In this …