A Perplexity-Based Method for Similar Languages Discrimination

Pablo Gamallo, Jose Ramom Pichel, Iñaki Alegria


Abstract
This article describes the system submitted by the Citius_Ixa_Imaxin team to the VarDial 2017 (DSL and GDI tasks). The strategy underlying our system is based on a language distance computed by means of model perplexity. The best model configuration we have tested is a voting system making use of several n-grams models of both words and characters, even if word unigrams turned out to be a very competitive model with reasonable results in the tasks we have participated. An error analysis has been performed in which we identified many test examples with no linguistic evidences to distinguish among the variants.
Anthology ID:
W17-1213
Volume:
Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial)
Month:
April
Year:
2017
Address:
Valencia, Spain
Editors:
Preslav Nakov, Marcos Zampieri, Nikola Ljubešić, Jörg Tiedemann, Shevin Malmasi, Ahmed Ali
Venue:
VarDial
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
109–114
Language:
URL:
https://aclanthology.org/W17-1213
DOI:
10.18653/v1/W17-1213
Bibkey:
Cite (ACL):
Pablo Gamallo, Jose Ramom Pichel, and Iñaki Alegria. 2017. A Perplexity-Based Method for Similar Languages Discrimination. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), pages 109–114, Valencia, Spain. Association for Computational Linguistics.
Cite (Informal):
A Perplexity-Based Method for Similar Languages Discrimination (Gamallo et al., VarDial 2017)
Copy Citation:
PDF:
https://aclanthology.org/W17-1213.pdf