Stochastic Tokenization with a Language Model for Neural Text Classification

Tatsuya Hiraoka, Hiroyuki Shindo, Yuji Matsumoto


Abstract
For unsegmented languages such as Japanese and Chinese, tokenization of a sentence has a significant impact on the performance of text classification. Sentences are usually segmented with words or subwords by a morphological analyzer or byte pair encoding and then encoded with word (or subword) representations for neural networks. However, segmentation is potentially ambiguous, and it is unclear whether the segmented tokens achieve the best performance for the target task. In this paper, we propose a method to simultaneously learn tokenization and text classification to address these problems. Our model incorporates a language model for unsupervised tokenization into a text classifier and then trains both models simultaneously. To make the model robust against infrequent tokens, we sampled segmentation for each sentence stochastically during training, which resulted in improved performance of text classification. We conducted experiments on sentiment analysis as a text classification task and show that our method achieves better performance than previous methods.
Anthology ID:
P19-1158
Volume:
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2019
Address:
Florence, Italy
Editors:
Anna Korhonen, David Traum, Lluís Màrquez
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1620–1629
Language:
URL:
https://aclanthology.org/P19-1158
DOI:
10.18653/v1/P19-1158
Bibkey:
Cite (ACL):
Tatsuya Hiraoka, Hiroyuki Shindo, and Yuji Matsumoto. 2019. Stochastic Tokenization with a Language Model for Neural Text Classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1620–1629, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
Stochastic Tokenization with a Language Model for Neural Text Classification (Hiraoka et al., ACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/P19-1158.pdf