<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="hronsky-keuleers-2024-tokenization">
    <titleInfo>
        <title>Tokenization via Language Modeling: the Role of Preceding Text</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Rastislav</namePart>
        <namePart type="family">Hronsky</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Emmanuel</namePart>
        <namePart type="family">Keuleers</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Second Workshop on Computation and Written Language (CAWL) @ LREC-COLING 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Kyle</namePart>
            <namePart type="family">Gorman</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Emily</namePart>
            <namePart type="family">Prud’hommeaux</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Brian</namePart>
            <namePart type="family">Roark</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Richard</namePart>
            <namePart type="family">Sproat</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>ELRA and ICCL</publisher>
            <place>
                <placeTerm type="text">Torino, Italia</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>While language models benefit immensely from their capacity to model large context (i.e., sequence of preceding tokens), the role of context is unclear in text tokenization, which is, in many cases, language model-driven to begin with. In this paper, we attempt to explore the role in three different writing systems and using three different text tokenization strategies (word-based, Morfessor, and BPE). In the first experiment, we examined how the size of context used for predicting the next token affects the ranking of the segmentation strategies i.t.o. language model surprisal. This effect was very writing system specific: minimal in case of English, and rank-reversing due to increased context size and token granularity in case of Turkish and Chinese. In the second experiment, we examined how context alters segmentation hypotheses when using language models to identify word boundaries. In this case, the effect was subtle: using context-aware, rather than context-free segment scores improved boundary recognition accuracy by up to 0.5%, once baseline effects were exploited.</abstract>
    <identifier type="citekey">hronsky-keuleers-2024-tokenization</identifier>
    <location>
        <url>https://aclanthology.org/2024.cawl-1.4</url>
    </location>
    <part>
        <date>2024-05</date>
        <extent unit="page">
            <start>23</start>
            <end>35</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Tokenization via Language Modeling: the Role of Preceding Text
%A Hronsky, Rastislav
%A Keuleers, Emmanuel
%Y Gorman, Kyle
%Y Prud’hommeaux, Emily
%Y Roark, Brian
%Y Sproat, Richard
%S Proceedings of the Second Workshop on Computation and Written Language (CAWL) @ LREC-COLING 2024
%D 2024
%8 May
%I ELRA and ICCL
%C Torino, Italia
%F hronsky-keuleers-2024-tokenization
%X While language models benefit immensely from their capacity to model large context (i.e., sequence of preceding tokens), the role of context is unclear in text tokenization, which is, in many cases, language model-driven to begin with. In this paper, we attempt to explore the role in three different writing systems and using three different text tokenization strategies (word-based, Morfessor, and BPE). In the first experiment, we examined how the size of context used for predicting the next token affects the ranking of the segmentation strategies i.t.o. language model surprisal. This effect was very writing system specific: minimal in case of English, and rank-reversing due to increased context size and token granularity in case of Turkish and Chinese. In the second experiment, we examined how context alters segmentation hypotheses when using language models to identify word boundaries. In this case, the effect was subtle: using context-aware, rather than context-free segment scores improved boundary recognition accuracy by up to 0.5%, once baseline effects were exploited.
%U https://aclanthology.org/2024.cawl-1.4
%P 23-35

Markdown (Informal)

[Tokenization via Language Modeling: the Role of Preceding Text](https://aclanthology.org/2024.cawl-1.4) (Hronsky & Keuleers, CAWL-WS 2024)

• Tokenization via Language Modeling: the Role of Preceding Text (Hronsky & Keuleers, CAWL-WS 2024)

ACL

• Rastislav Hronsky and Emmanuel Keuleers. 2024. Tokenization via Language Modeling: the Role of Preceding Text. In Proceedings of the Second Workshop on Computation and Written Language (CAWL) @ LREC-COLING 2024, pages 23–35, Torino, Italia. ELRA and ICCL.