Yiran Zhong

The Devil in Linear Transformer

2022 · arXiv (Cornell University)

Linear transformers aim to reduce the quadratic space-time complexity of vanilla transformers. However, they usually suffer from degraded performances on various tasks and corpus. In this paper, we examine existing kernel-based linear transformers and identify …

Scaling Laws for Linear Complexity Language Models

2024 · arXiv (Cornell University)

The interest in linear complexity models for large language models is on the rise, although their scaling capacity remains uncertain. In this study, we present the scaling laws for linear complexity language models to establish …