LanideNN: Multilingual Language Identification on Character Window

Tom Kocmi, Ondřej Bojar


Abstract
In language identification, a common first step in natural language processing, we want to automatically determine the language of some input text. Monolingual language identification assumes that the given document is written in one language. In multilingual language identification, the document is usually in two or three languages and we just want their names. We aim one step further and propose a method for textual language identification where languages can change arbitrarily and the goal is to identify the spans of each of the languages. Our method is based on Bidirectional Recurrent Neural Networks and it performs well in monolingual and multilingual language identification tasks on six datasets covering 131 languages. The method keeps the accuracy also for short documents and across domains, so it is ideal for off-the-shelf use without preparation of training data.
Anthology ID:
E17-1087
Volume:
Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers
Month:
April
Year:
2017
Address:
Valencia, Spain
Editors:
Mirella Lapata, Phil Blunsom, Alexander Koller
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
927–936
Language:
URL:
https://aclanthology.org/E17-1087
DOI:
Bibkey:
Cite (ACL):
Tom Kocmi and Ondřej Bojar. 2017. LanideNN: Multilingual Language Identification on Character Window. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 927–936, Valencia, Spain. Association for Computational Linguistics.
Cite (Informal):
LanideNN: Multilingual Language Identification on Character Window (Kocmi & Bojar, EACL 2017)
Copy Citation:
PDF:
https://aclanthology.org/E17-1087.pdf
Code
 tomkocmi/LanideNN