The Korean Morphologically Tight-Fitting Tokenizer for Noisy User-Generated Texts

Sangah Lee, Hyopil Shin


Abstract
User-generated texts include various types of stylistic properties, or noises. Such texts are not properly processed by existing morpheme analyzers or language models based on formal texts such as encyclopedias or news articles. In this paper, we propose a simple morphologically tight-fitting tokenizer (K-MT) that can better process proper nouns, coinages, and internet slang among other types of noise in Korean user-generated texts. We tested our tokenizer by performing classification tasks on Korean user-generated movie reviews and hate speech datasets, and the Korean Named Entity Recognition dataset. Through our tests, we found that K-MT is better fit to process internet slangs, proper nouns, and coinages, compared to a morpheme analyzer and a character-level WordPiece tokenizer.
Anthology ID:
2021.wnut-1.45
Volume:
Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021)
Month:
November
Year:
2021
Address:
Online
Editors:
Wei Xu, Alan Ritter, Tim Baldwin, Afshin Rahimi
Venue:
WNUT
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
410–416
Language:
URL:
https://aclanthology.org/2021.wnut-1.45
DOI:
10.18653/v1/2021.wnut-1.45
Bibkey:
Cite (ACL):
Sangah Lee and Hyopil Shin. 2021. The Korean Morphologically Tight-Fitting Tokenizer for Noisy User-Generated Texts. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 410–416, Online. Association for Computational Linguistics.
Cite (Informal):
The Korean Morphologically Tight-Fitting Tokenizer for Noisy User-Generated Texts (Lee & Shin, WNUT 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.wnut-1.45.pdf