Masked Part-Of-Speech Model: Does Modeling Long Context Help Unsupervised POS-tagging?

Xiang Zhou, Shiyue Zhang, Mohit Bansal


Abstract
Previous Part-Of-Speech (POS) induction models usually assume certain independence assumptions (e.g., Markov, unidirectional, local dependency) that do not hold in real languages. For example, the subject-verb agreement can be both long-term and bidirectional. To facilitate flexible dependency modeling, we propose a Masked Part-of-Speech Model (MPoSM), inspired by the recent success of Masked Language Models (MLM). MPoSM can model arbitrary tag dependency and perform POS induction through the objective of masked POS reconstruction. We achieve competitive results on both the English Penn WSJ dataset as well as the universal treebank containing 10 diverse languages. Though modeling the long-term dependency should ideally help this task, our ablation study shows mixed trends in different languages. To better understand this phenomenon, we design a novel synthetic experiment that can specifically diagnose the model’s ability to learn tag agreement. Surprisingly, we find that even strong baselines fail to solve this problem consistently in a very simplified setting: the agreement between adjacent words. Nonetheless, MPoSM achieves overall better performance. Lastly, we conduct a detailed error analysis to shed light on other remaining challenges.
Anthology ID:
2022.naacl-main.81
Volume:
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
July
Year:
2022
Address:
Seattle, United States
Editors:
Marine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1099–1114
Language:
URL:
https://aclanthology.org/2022.naacl-main.81
DOI:
10.18653/v1/2022.naacl-main.81
Bibkey:
Cite (ACL):
Xiang Zhou, Shiyue Zhang, and Mohit Bansal. 2022. Masked Part-Of-Speech Model: Does Modeling Long Context Help Unsupervised POS-tagging?. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1099–1114, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
Masked Part-Of-Speech Model: Does Modeling Long Context Help Unsupervised POS-tagging? (Zhou et al., NAACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.naacl-main.81.pdf
Video:
 https://aclanthology.org/2022.naacl-main.81.mp4
Code
 owenzx/mposm