Automatically Extracting Variant-Normalization Pairs for Japanese Text Normalization

Itsumi Saito, Kyosuke Nishida, Kugatsu Sadamitsu, Kuniko Saito, Junji Tomita


Abstract
Social media texts, such as tweets from Twitter, contain many types of non-standard tokens, and the number of normalization approaches for handling such noisy text has been increasing. We present a method for automatically extracting pairs of a variant word and its normal form from unsegmented text on the basis of a pair-wise similarity approach. We incorporated the acquired variant-normalization pairs into Japanese morphological analysis. The experimental results show that our method can extract widely covered variants from large Twitter data and improve the recall of normalization without degrading the overall accuracy of Japanese morphological analysis.
Anthology ID:
I17-1094
Volume:
Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Month:
November
Year:
2017
Address:
Taipei, Taiwan
Editors:
Greg Kondrak, Taro Watanabe
Venue:
IJCNLP
SIG:
Publisher:
Asian Federation of Natural Language Processing
Note:
Pages:
937–946
Language:
URL:
https://aclanthology.org/I17-1094
DOI:
Bibkey:
Cite (ACL):
Itsumi Saito, Kyosuke Nishida, Kugatsu Sadamitsu, Kuniko Saito, and Junji Tomita. 2017. Automatically Extracting Variant-Normalization Pairs for Japanese Text Normalization. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 937–946, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Cite (Informal):
Automatically Extracting Variant-Normalization Pairs for Japanese Text Normalization (Saito et al., IJCNLP 2017)
Copy Citation:
PDF:
https://aclanthology.org/I17-1094.pdf