Kazuo Hara
Author directory2026
Beyond Junk Tokens: Subsumed Tokens in Japanese Subword Tokenization
Toshiki Yamaguchi | Kazuo Hara | Ikumi Suzuki
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Toshiki Yamaguchi | Kazuo Hara | Ikumi Suzuki
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Subword tokenization is widely used to address the out-of-vocabulary problem in natural language processing. However, frequency-based subword tokenization can produce artifacts in the resulting vocabulary. Beyond the well-known junk tokens (meaningless fragments), we introduce the notion of subsumed tokens, valid words whose corpus frequencies are suppressed by longer tokens that contain them. Together with fragment tokens, these are referred to as overlapping tokens. We propose a post-processing method that reduces fragment tokens and restores subsumed tokens. The method searches over possible binary segmentations of character sequences and selects tokenizations that maximize corpus frequency, while optionally assigning bonuses to word spans identified by a morphological analyzer. As a post-processing step, the method can be applied to existing tokenizers such as Byte Pair Encoding (BPE) and the Unigram Language Model (ULM) without rebuilding their vocabularies. Although evaluated on Japanese, the approach may also be applicable to other languages given an appropriate morphological analyzer. Experiments on the Wiki-40B Japanese corpus show that our method reduces overlapping tokens, leading to more reliable statistics for valid words, and improves representation quality while preserving downstream performance across tasks.
2015
Coordination-Aware Dependency Parsing (Preliminary Report)
Akifumi Yoshimoto | Kazuo Hara | Masashi Shimbo | Yuji Matsumoto
Proceedings of the 14th International Conference on Parsing Technologies
Akifumi Yoshimoto | Kazuo Hara | Masashi Shimbo | Yuji Matsumoto
Proceedings of the 14th International Conference on Parsing Technologies
2013
Centering Similarity Measures to Reduce Hubs
Ikumi Suzuki | Kazuo Hara | Masashi Shimbo | Marco Saerens | Kenji Fukumizu
Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing
Ikumi Suzuki | Kazuo Hara | Masashi Shimbo | Marco Saerens | Kenji Fukumizu
Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing
2012
Walk-based Computation of Contextual Word Similarity
Kazuo Hara | Ikumi Suzuki | Masashi Shimbo | Yuji Matsumoto
Proceedings of COLING 2012
Kazuo Hara | Ikumi Suzuki | Masashi Shimbo | Yuji Matsumoto
Proceedings of COLING 2012
2009
Bypassed alignment graph for learning coordination in Japanese sentences
Hideharu Okuma | Kazuo Hara | Masashi Shimbo | Yuji Matsumoto
Proceedings of the ACL-IJCNLP 2009 Conference Short Papers
Hideharu Okuma | Kazuo Hara | Masashi Shimbo | Yuji Matsumoto
Proceedings of the ACL-IJCNLP 2009 Conference Short Papers
Coordinate Structure Analysis with Global Structural Constraints and Alignment-Based Local Features
Kazuo Hara | Masashi Shimbo | Hideharu Okuma | Yuji Matsumoto
Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP
Kazuo Hara | Masashi Shimbo | Hideharu Okuma | Yuji Matsumoto
Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP