Daniil Chernyshev

Impact of Tokenization on LLaMa Russian Adaptation

2023 · arXiv (Cornell University)

Latest instruction-tuned large language models (LLM) show great results on various tasks, however, they often face performance degradation for non-English input. There is evidence that the reason lies in inefficient tokenization caused by low language …