<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="hiraoka-2026-corpus">
    <titleInfo>
        <title>Corpus-Dependent Subcharacter Encoding via HMM-Guided Code Assignment</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Tatsuya</namePart>
        <namePart type="family">Hiraoka</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Maria</namePart>
            <namePart type="family">Liakata</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Viviane</namePart>
            <namePart type="given">P</namePart>
            <namePart type="family">Moreira</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jiajun</namePart>
            <namePart type="family">Zhang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">David</namePart>
            <namePart type="family">Jurgens</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">San Diego, California, United States</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-390-6</identifier>
    </relatedItem>
    <abstract>We propose a corpus-dependent alternative to byte encoding that learns fixed-length atomic codes for characters directly from text, which we refer to as Latom (Learned Atom-based Encoding).We instantiate this framework by training an HMM on N-repeated character sequences to estimate “atom” posteriors, followed by a Hungarian assignment yielding a globally optimal one-to-one character-code mapping.Across 14 languages, the encodings improve intrinsic metrics, including token counts after subword tokenization and bigram perplexity, with appropriate code lengths.On Amazon Reviews in six languages, Latom improves text classification accuracy and reduces decoding errors in language model generation.Overall, these results demonstrate that character encodings can be learned from corpus statistics while remaining reversible and compatible with standard tokenization pipelines.</abstract>
    <identifier type="citekey">hiraoka-2026-corpus</identifier>
    <identifier type="doi">10.18653/v1/2026.acl-long.1596</identifier>
    <location>
        <url>https://aclanthology.org/2026.acl-long.1596/</url>
    </location>
    <part>
        <date>2026-07</date>
        <extent unit="page">
            <start>34569</start>
            <end>34593</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Corpus-Dependent Subcharacter Encoding via HMM-Guided Code Assignment
%A Hiraoka, Tatsuya
%Y Liakata, Maria
%Y Moreira, Viviane P.
%Y Zhang, Jiajun
%Y Jurgens, David
%S Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
%D 2026
%8 July
%I Association for Computational Linguistics
%C San Diego, California, United States
%@ 979-8-89176-390-6
%F hiraoka-2026-corpus
%X We propose a corpus-dependent alternative to byte encoding that learns fixed-length atomic codes for characters directly from text, which we refer to as Latom (Learned Atom-based Encoding).We instantiate this framework by training an HMM on N-repeated character sequences to estimate “atom” posteriors, followed by a Hungarian assignment yielding a globally optimal one-to-one character-code mapping.Across 14 languages, the encodings improve intrinsic metrics, including token counts after subword tokenization and bigram perplexity, with appropriate code lengths.On Amazon Reviews in six languages, Latom improves text classification accuracy and reduces decoding errors in language model generation.Overall, these results demonstrate that character encodings can be learned from corpus statistics while remaining reversible and compatible with standard tokenization pipelines.
%R 10.18653/v1/2026.acl-long.1596
%U https://aclanthology.org/2026.acl-long.1596/
%U https://doi.org/10.18653/v1/2026.acl-long.1596
%P 34569-34593

Markdown (Informal)

[Corpus-Dependent Subcharacter Encoding via HMM-Guided Code Assignment](https://aclanthology.org/2026.acl-long.1596/) (Hiraoka, ACL 2026)

• Corpus-Dependent Subcharacter Encoding via HMM-Guided Code Assignment (Hiraoka, ACL 2026)

ACL

• Tatsuya Hiraoka. 2026. Corpus-Dependent Subcharacter Encoding via HMM-Guided Code Assignment. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 34569–34593, San Diego, California, United States. Association for Computational Linguistics.