<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="sivakumaran-etal-2026-comparative">
    <titleInfo>
        <title>Comparative Analysis of Tokenizers in Tamil Text Classification in Low Resource Settings</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Gokulan</namePart>
        <namePart type="family">Sivakumaran</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Randil</namePart>
        <namePart type="family">Pushpananda</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">E</namePart>
        <namePart type="given">R</namePart>
        <namePart type="given">A</namePart>
        <namePart type="given">D</namePart>
        <namePart type="family">Bandara</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Kengatharaiyer</namePart>
            <namePart type="family">Sarveswaran</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ashwini</namePart>
            <namePart type="family">Vaidya</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>ELRA Language Resources Association (ELRA)</publisher>
            <place>
                <placeTerm type="text">Palma de Mallorca, Spain</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Tokenization is crucial in NLP, influencing performance for morphologically rich, low resource languages like Tamil. This study comprehensively analyzes WordPiece, SentencePiece, and Byte-Level Byte Pair Encoding (BBPE) for Tamil text classification. We assess tokenization efficiency using metrics including token count, fragmentation, OOV rate, and compression ratio. Additionally, we analyze downstream impact through Tamil news title classification using a custom lightweight BERT based Transformer architecture. Tokenizers were pretrained on a 5.45 GB Tamil Corpus and evaluated on a Kaggle Tamil News Dataset. Results indicate WordPiece and SentencePiece outperform BBPE in efficiency and accuracy. While BBPE eliminates OOV words, excessive fragmentation hinders model learning. Increasing vocabulary size improves WordPiece and SentencePiece but not BBPE. Misclassification analysis highlights overfragmentation challenges. This study contributes to Tamil NLP by comparing tokenizers, aiding researchers in selecting appropriate strategies for agglutinative languages.</abstract>
    <identifier type="citekey">sivakumaran-etal-2026-comparative</identifier>
    <identifier type="doi">10.63317/5p78kf96x2jw</identifier>
    <location>
        <url>https://aclanthology.org/2026.chipsal-1.19/</url>
    </location>
    <part>
        <date>2026-05</date>
        <extent unit="page">
            <start>198</start>
            <end>208</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Comparative Analysis of Tokenizers in Tamil Text Classification in Low Resource Settings
%A Sivakumaran, Gokulan
%A Pushpananda, Randil
%A Bandara, E. R. A. D.
%Y Sarveswaran, Kengatharaiyer
%Y Vaidya, Ashwini
%S Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)
%D 2026
%8 May
%I ELRA Language Resources Association (ELRA)
%C Palma de Mallorca, Spain
%F sivakumaran-etal-2026-comparative
%X Tokenization is crucial in NLP, influencing performance for morphologically rich, low resource languages like Tamil. This study comprehensively analyzes WordPiece, SentencePiece, and Byte-Level Byte Pair Encoding (BBPE) for Tamil text classification. We assess tokenization efficiency using metrics including token count, fragmentation, OOV rate, and compression ratio. Additionally, we analyze downstream impact through Tamil news title classification using a custom lightweight BERT based Transformer architecture. Tokenizers were pretrained on a 5.45 GB Tamil Corpus and evaluated on a Kaggle Tamil News Dataset. Results indicate WordPiece and SentencePiece outperform BBPE in efficiency and accuracy. While BBPE eliminates OOV words, excessive fragmentation hinders model learning. Increasing vocabulary size improves WordPiece and SentencePiece but not BBPE. Misclassification analysis highlights overfragmentation challenges. This study contributes to Tamil NLP by comparing tokenizers, aiding researchers in selecting appropriate strategies for agglutinative languages.
%R 10.63317/5p78kf96x2jw
%U https://aclanthology.org/2026.chipsal-1.19/
%U https://doi.org/10.63317/5p78kf96x2jw
%P 198-208

Markdown (Informal)

[Comparative Analysis of Tokenizers in Tamil Text Classification in Low Resource Settings](https://aclanthology.org/2026.chipsal-1.19/) (Sivakumaran et al., CHiPSAL 2026)

• Comparative Analysis of Tokenizers in Tamil Text Classification in Low Resource Settings (Sivakumaran et al., CHiPSAL 2026)

ACL

• Gokulan Sivakumaran, Randil Pushpananda, and ERAD Bandara. 2026. Comparative Analysis of Tokenizers in Tamil Text Classification in Low Resource Settings. In Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026), pages 198–208, Palma de Mallorca, Spain. ELRA Language Resources Association (ELRA).