Jonne Sälevä

What changes when you randomly choose BPE merge operations? Not much.

2023

We introduce two simple randomized variants of byte pair encoding (BPE) and explore whether randomizing the selection of merge operations substantially affects a downstream machine translation task. We focus on translation into morphologically rich languages, …

OpenNER 1.0: Standardized Open-Access Named Entity Recognition Datasets in 50+ Languages

2024 · arXiv (Cornell University)

We present OpenNER 1.0, a standardized collection of openly-available named entity recognition (NER) datasets. OpenNER contains 36 NER corpora that span 52 languages, human-annotated in varying named entity ontologies. We correct annotation format issues, standardize …