Conglong Li

Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformers

2022 · arXiv (Cornell University)

Large-scale transformer models have become the de-facto architectures for various machine learning applications, e.g., CV and NLP. However, those large models also introduce prohibitive training costs. To mitigate this issue, we propose a novel random …