Fumbling in Babel: An Investigation into ChatGPT’s Language Identification Ability

Wei-Rui Chen, Ife Adebara, Khai Doan, Qisheng Liao, Muhammad Abdul-Mageed


Abstract
ChatGPT has recently emerged as a powerful NLP tool that can carry out a variety of tasks. However, the range of languages ChatGPT can handle remains largely a mystery. To uncover which languages ChatGPT ‘knows’, we investigate its language identification (LID) abilities. For this purpose, we compile Babel-670, a benchmark comprising 670 languages representing 23 language families spoken in five continents. Languages in Babel-670 run the gamut from the very high-resource to the very low-resource. We then study ChatGPT’s (both GPT-3.5 and GPT-4) ability to (i) identify language names and language codes (ii) under zero- and few-shot conditions (iii) with and without provision of a label set. When compared to smaller finetuned LID tools, we find that ChatGPT lags behind. For example, it has poor performance on African languages. We conclude that current large language models would benefit from further development before they can sufficiently serve diverse communities.
Anthology ID:
2024.findings-naacl.274
Volume:
Findings of the Association for Computational Linguistics: NAACL 2024
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4387–4413
Language:
URL:
https://aclanthology.org/2024.findings-naacl.274
DOI:
Bibkey:
Cite (ACL):
Wei-Rui Chen, Ife Adebara, Khai Doan, Qisheng Liao, and Muhammad Abdul-Mageed. 2024. Fumbling in Babel: An Investigation into ChatGPT’s Language Identification Ability. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4387–4413, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Fumbling in Babel: An Investigation into ChatGPT’s Language Identification Ability (Chen et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-naacl.274.pdf
Copyright:
 2024.findings-naacl.274.copyright.pdf