Code-Switched Language Identification is Harder Than You Think

Laurie Burchell, Alexandra Birch, Robert Thompson, Kenneth Heafield


Abstract
Code switching (CS) is a very common phenomenon in written and spoken communication, but is handled poorly by many NLP applications. Looking to the application of building CS corpora, we explore CS language identification for corpus building. We make the task more realistic by scaling it to more languages and considering models with simpler architectures for faster inference. We also reformulate the task as a sentence-level multi-label tagging problem to make it more tractable. Having defined the task, we investigate three reasonable architectures for this task and define metrics which better reflect desired performance. We present empirical evidence that no current approach is adequate, and finally provide recommendations for future work in this area.
Anthology ID:
2024.eacl-long.38
Volume:
Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
March
Year:
2024
Address:
St. Julian’s, Malta
Editors:
Yvette Graham, Matthew Purver
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
646–658
Language:
URL:
https://aclanthology.org/2024.eacl-long.38
DOI:
Bibkey:
Cite (ACL):
Laurie Burchell, Alexandra Birch, Robert Thompson, and Kenneth Heafield. 2024. Code-Switched Language Identification is Harder Than You Think. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 646–658, St. Julian’s, Malta. Association for Computational Linguistics.
Cite (Informal):
Code-Switched Language Identification is Harder Than You Think (Burchell et al., EACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.eacl-long.38.pdf