Jianan Wang

LipsFormer: Introducing Lipschitz Continuity to Vision Transformers

2023 · arXiv (Cornell University)

We present a Lipschitz continuous Transformer, called LipsFormer, to pursue training stability both theoretically and empirically for Transformer-based models. In contrast to previous practical tricks that address training instability by learning rate warmup, layer normalization, …