<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="yamaguchi-etal-2026-beyond">
    <titleInfo>
        <title>Beyond Junk Tokens: Subsumed Tokens in Japanese Subword Tokenization</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Toshiki</namePart>
        <namePart type="family">Yamaguchi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kazuo</namePart>
        <namePart type="family">Hara</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ikumi</namePart>
        <namePart type="family">Suzuki</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-459-0</identifier>
    </relatedItem>
    <abstract>Subword tokenization is widely used to address the out-of-vocabulary problem in natural language processing. However, frequency-based subword tokenization can produce artifacts in the resulting vocabulary. Beyond the well-known junk tokens (meaningless fragments), we introduce the notion of subsumed tokens, valid words whose corpus frequencies are suppressed by longer tokens that contain them. Together with fragment tokens, these are referred to as overlapping tokens. We propose a post-processing method that reduces fragment tokens and restores subsumed tokens. The method searches over possible binary segmentations of character sequences and selects tokenizations that maximize corpus frequency, while optionally assigning bonuses to word spans identified by a morphological analyzer. As a post-processing step, the method can be applied to existing tokenizers such as Byte Pair Encoding (BPE) and the Unigram Language Model (ULM) without rebuilding their vocabularies. Although evaluated on Japanese, the approach may also be applicable to other languages given an appropriate morphological analyzer. Experiments on the Wiki-40B Japanese corpus show that our method reduces overlapping tokens, leading to more reliable statistics for valid words, and improves representation quality while preserving downstream performance across tasks.</abstract>
    <identifier type="citekey">yamaguchi-etal-2026-beyond</identifier>
    <location>
        <url>https://aclanthology.org/2026.emnlp-main.153/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>3534</start>
            <end>3545</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Beyond Junk Tokens: Subsumed Tokens in Japanese Subword Tokenization
%A Yamaguchi, Toshiki
%A Hara, Kazuo
%A Suzuki, Ikumi
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F yamaguchi-etal-2026-beyond
%X Subword tokenization is widely used to address the out-of-vocabulary problem in natural language processing. However, frequency-based subword tokenization can produce artifacts in the resulting vocabulary. Beyond the well-known junk tokens (meaningless fragments), we introduce the notion of subsumed tokens, valid words whose corpus frequencies are suppressed by longer tokens that contain them. Together with fragment tokens, these are referred to as overlapping tokens. We propose a post-processing method that reduces fragment tokens and restores subsumed tokens. The method searches over possible binary segmentations of character sequences and selects tokenizations that maximize corpus frequency, while optionally assigning bonuses to word spans identified by a morphological analyzer. As a post-processing step, the method can be applied to existing tokenizers such as Byte Pair Encoding (BPE) and the Unigram Language Model (ULM) without rebuilding their vocabularies. Although evaluated on Japanese, the approach may also be applicable to other languages given an appropriate morphological analyzer. Experiments on the Wiki-40B Japanese corpus show that our method reduces overlapping tokens, leading to more reliable statistics for valid words, and improves representation quality while preserving downstream performance across tasks.
%U https://aclanthology.org/2026.emnlp-main.153/
%P 3534-3545

Markdown (Informal)

[Beyond Junk Tokens: Subsumed Tokens in Japanese Subword Tokenization](https://aclanthology.org/2026.emnlp-main.153/) (Yamaguchi et al., EMNLP 2026)

• Beyond Junk Tokens: Subsumed Tokens in Japanese Subword Tokenization (Yamaguchi et al., EMNLP 2026)

ACL

• Toshiki Yamaguchi, Kazuo Hara, and Ikumi Suzuki. 2026. Beyond Junk Tokens: Subsumed Tokens in Japanese Subword Tokenization. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 3534–3545, Budapest, Hungary. Association for Computational Linguistics.