A Probabilistic Toolkit for Multi-grained Word Segmentation in Chinese

Xi Ma, Yang Hou, Xuebin Wang, Zhenghua Li


Abstract
It is practically useful to provide consistent and reliable word segmentation results from different criteria at the same time, which is formulated as the multi-grained word segmentation (MWS) task. This paper describes a probabilistic toolkit for MWS in Chinese. We propose a new MWS approach based on the standard MTL framework. We adopt semi-Markov CRF for single-grained word segmentation (SWS), which can produce marginal probabilities of words during inference. For sentences that contain conflicts among SWS results, we employ the CKY decoding algorithm to resolve conflicts.Our resulting MWS tree can provide the criteria information of words, along with the probabilities. Moreover, we follow the works in SWS, and propose a simple strategy to exploit naturally annotated data for MWS, leading to substantial improvement of MWS performance in the cross-domain scenario.
Anthology ID:
2025.coling-demos.9
Volume:
Proceedings of the 31st International Conference on Computational Linguistics: System Demonstrations
Month:
January
Year:
2025
Address:
Abu Dhabi, UAE
Editors:
Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, Steven Schockaert, Brodie Mather, Mark Dras
Venue:
COLING
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
83–90
Language:
URL:
https://aclanthology.org/2025.coling-demos.9/
DOI:
Bibkey:
Cite (ACL):
Xi Ma, Yang Hou, Xuebin Wang, and Zhenghua Li. 2025. A Probabilistic Toolkit for Multi-grained Word Segmentation in Chinese. In Proceedings of the 31st International Conference on Computational Linguistics: System Demonstrations, pages 83–90, Abu Dhabi, UAE. Association for Computational Linguistics.
Cite (Informal):
A Probabilistic Toolkit for Multi-grained Word Segmentation in Chinese (Ma et al., COLING 2025)
Copy Citation:
PDF:
https://aclanthology.org/2025.coling-demos.9.pdf