Improved Neural Word Segmentation for Standard Tibetan

Collin J. Brown


Abstract
As Tibetan is traditionally not written with word delimiters, various means of word segmentation are necessary to prepare data for downstream tasks. Neural word segmentation has proven a successful means of parsing Tibetan text, but current performance lags behind that of neural word segmenters in other languages, such as Chinese or Japanese, and even behind languages with relatively similar orthographic structures, such as Vietnamese or Thai. We apply methods that have proven useful for these latter two languages , in addition to Classical Tibetan, toward the development of a neural word segmenter with the goal of raising the peak performance of Tibetan neural word segmentation to a level comparable to that reached for orthographically similar languages.
Anthology ID:
2024.eurali-1.2
Volume:
Proceedings of the 2nd Workshop on Resources and Technologies for Indigenous, Endangered and Lesser-resourced Languages in Eurasia (EURALI) @ LREC-COLING 2024
Month:
May
Year:
2024
Address:
Torino, Italia
Editors:
Atul Kr. Ojha, Sina Ahmadi, Silvie Cinková, Theodorus Fransen, Chao-Hong Liu, John P. McCrae
Venues:
EURALI | WS
SIG:
Publisher:
ELRA and ICCL
Note:
Pages:
12–17
Language:
URL:
https://aclanthology.org/2024.eurali-1.2
DOI:
Bibkey:
Cite (ACL):
Collin J. Brown. 2024. Improved Neural Word Segmentation for Standard Tibetan. In Proceedings of the 2nd Workshop on Resources and Technologies for Indigenous, Endangered and Lesser-resourced Languages in Eurasia (EURALI) @ LREC-COLING 2024, pages 12–17, Torino, Italia. ELRA and ICCL.
Cite (Informal):
Improved Neural Word Segmentation for Standard Tibetan (Brown, EURALI-WS 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.eurali-1.2.pdf