preprint
وصول مفتوح
Akshara: Script-Aware, Grapheme-Cluster-Correct Array Operations for Indic Text
Research footprint
At a glance
- الاستشهادات
- 0
- المراجع
- 0
- Comments
- 0
Paper overview
Abstract
pandas, Polars, and Apache Arrow represent string columns as sequences of Unicode codepoints. For the nine Brahmic abugida scripts (Devanagari, Bengali, Gurmukhi, Gujarati, Odia, Tamil, Telugu, Kannada, Malayalam), a single user-perceived unit — an akshara, or orthographic syllable — is frequently encoded as a cluster of several codepoints, causing len() and slicing to silently corrupt conjuncts. This report introduces Akshara, a pure Python/NumPy library providing script-aware, grapheme-cluster-correct array operations — linear-time akshara segmentation, vectorized batch operations, and corpus pipeline integration — validated against 9,108 real FLORES-200 sentences across all nine target scripts with zero failures.
Record transparency
Publication details
- DOI
- 10.5281/zenodo.21697082
- OpenAlex
- W7171843930
- Document type
- preprint
- Language
- EN
- Source
- Zenodo (CERN European Organization for Nuclear Research)
- Last metadata update
Comments
تسجيل الدخول للانضمام إلى النقاش.