preprint وصول مفتوح

Akshara: Script-Aware, Grapheme-Cluster-Correct Array Operations for Indic Text

  • Zenodo (CERN European Organization for Nuclear Research)
  • European Organization for Nuclear Research
Research footprint

At a glance

الاستشهادات
0
المراجع
0
Comments
0
Paper overview

Abstract

pandas, Polars, and Apache Arrow represent string columns as sequences of Unicode codepoints. For the nine Brahmic abugida scripts (Devanagari, Bengali, Gurmukhi, Gujarati, Odia, Tamil, Telugu, Kannada, Malayalam), a single user-perceived unit — an akshara, or orthographic syllable — is frequently encoded as a cluster of several codepoints, causing len() and slicing to silently corrupt conjuncts. This report introduces Akshara, a pure Python/NumPy library providing script-aware, grapheme-cluster-correct array operations — linear-time akshara segmentation, vectorized batch operations, and corpus pipeline integration — validated against 9,108 real FLORES-200 sentences across all nine target scripts with zero failures.

Record transparency

Publication details

DOI
10.5281/zenodo.21697082
OpenAlex
W7171843930
Document type
preprint
Language
EN
Source
Zenodo (CERN European Organization for Nuclear Research)
Last metadata update
المجتمع

Comments

تسجيل الدخول للانضمام إلى النقاش.

  1. لا توجد تعليقات بعد. ابدأ النقاش.