Siyu Ren

Context Compression for Auto-regressive Transformers with Sentinel Tokens

2023

The quadratic complexity of the attention module makes it gradually become the bulk of compute in Transformer-based LLMs during generation. Moreover, the excessive key-value cache that arises when dealing with long inputs also brings severe …