Word Level Language Identification and Back-Transliteration

In this paper, I describe a Rule based and List-Searching system for Word-Level Language Identification and Named Entity recognition in bilingual text. My method uses dictionary search, rules for LI and CRF++, character n-gram for NER. The model does not use any Language specific rules, therefore can easily be replicated on most languages having mixed pair with English. The model also does back-transliteration of words into native language script and recognizes named entity. The model performance is carried on the test sets provided by the shared task on language Identification for English Hindi (En-Hi) Pair, Microsoft Research India. The experimental results show a consistent performance with high precision.