Federico Barbero

Round and Round We Go! What makes Rotary Positional Encodings useful?

2024 · arXiv (Cornell University)

Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary …