Toshinobu Ogiso

Long Unit Word Tokenization and Bunsetsu Segmentation of Historical Japanese

2024

In Japanese, "bunsetsu" is the natural minimal phrase of a sentence; it serves as a natural boundary of a sentence for native speakers rather than words, and thus grammatical analysis in Japanese linguistics commonly operates …