How does Dictionary Size Influence Performance of Vietnamese Word Segmentation?

Wuying Liu, Lin Wang


Abstract
Vietnamese word segmentation (VWS) is a challenging basic issue for natural language processing. This paper addresses the problem of how does dictionary size influence VWS performance, proposes two novel measures: square overlap ratio (SOR) and relaxed square overlap ratio (RSOR), and validates their effectiveness. The SOR measure is the product of dictionary overlap ratio and corpus overlap ratio, and the RSOR measure is the relaxed version of SOR measure under an unsupervised condition. The two measures both indicate the suitable degree between segmentation dictionary and object corpus waiting for segmentation. The experimental results show that the more suitable, neither smaller nor larger, dictionary size is better to achieve the state-of-the-art performance for dictionary-based Vietnamese word segmenters.
Anthology ID:
L16-1172
Volume:
Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)
Month:
May
Year:
2016
Address:
Portorož, Slovenia
Editors:
Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
1079–1083
Language:
URL:
https://aclanthology.org/L16-1172
DOI:
Bibkey:
Cite (ACL):
Wuying Liu and Lin Wang. 2016. How does Dictionary Size Influence Performance of Vietnamese Word Segmentation?. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1079–1083, Portorož, Slovenia. European Language Resources Association (ELRA).
Cite (Informal):
How does Dictionary Size Influence Performance of Vietnamese Word Segmentation? (Liu & Wang, LREC 2016)
Copy Citation:
PDF:
https://aclanthology.org/L16-1172.pdf