Anton Kolonin

Unsupervised Tokenization Learning

2022 · arXiv (Cornell University)

In the presented study, we discover that the so-called "transition freedom" metric appears superior for unsupervised tokenization purposes in comparison to statistical metrics such as mutual information and conditional probability, providing F-measure scores in range …