Huiyin Xue

HashFormers: Towards Vocabulary-independent Pre-trained Transformers

2022

Transformer-based pre-trained language models are vocabulary-dependent, mapping by default each token to its corresponding embedding. This one-to-one mapping results into embedding matrices that occupy a lot of memory (i.e. millions of parameters) and grow linearly …