Detection of Similar Languages and Dialects Using Deep Supervised Autoencoder

Shantipriya Parida, Esau Villatoro-Tello, Sajit Kumar, Maël Fabien, Petr Motlicek


Abstract
Language detection is considered a difficult task especially for similar languages, varieties, and dialects. With the growing number of online content in different languages, the need for reliable and robust language detection tools also increased. In this work, we use supervised autoencoders with a bayesian optimizer for language detection and highlights its efficiency in detecting similar languages with dialect variance in comparison to other state-of-the-art techniques. We evaluated our approach on multiple datasets (Ling10, Discriminating between Similar Language (DSL), and Indo-Aryan Language Identification (ILI)). Obtained results demonstrate that SAE are higly effective in detecting languages, up to a 100% accuracy in the Ling10. Similarly, we obtain a competitive performance in identifying similar languages, and dialects, 92% and 85% for DSL ans ILI datasets respectively.
Anthology ID:
2020.icon-main.49
Volume:
Proceedings of the 17th International Conference on Natural Language Processing (ICON)
Month:
December
Year:
2020
Address:
Indian Institute of Technology Patna, Patna, India
Editors:
Pushpak Bhattacharyya, Dipti Misra Sharma, Rajeev Sangal
Venue:
ICON
SIG:
Publisher:
NLP Association of India (NLPAI)
Note:
Pages:
362–367
Language:
URL:
https://aclanthology.org/2020.icon-main.49
DOI:
Bibkey:
Cite (ACL):
Shantipriya Parida, Esau Villatoro-Tello, Sajit Kumar, Maël Fabien, and Petr Motlicek. 2020. Detection of Similar Languages and Dialects Using Deep Supervised Autoencoder. In Proceedings of the 17th International Conference on Natural Language Processing (ICON), pages 362–367, Indian Institute of Technology Patna, Patna, India. NLP Association of India (NLPAI).
Cite (Informal):
Detection of Similar Languages and Dialects Using Deep Supervised Autoencoder (Parida et al., ICON 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.icon-main.49.pdf