<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="lotz-etal-2026-getting">
    <titleInfo>
        <title>Getting Close to Cloze: Investigating Language Model and Human Cloze-test Performance in Afrikaans</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Susan</namePart>
        <namePart type="family">Lotz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Rik</namePart>
        <namePart type="family">van Noord</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Gertjan</namePart>
        <namePart type="family">van Noord</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of Resources for African Indigenous Languages (RAIL) 2026 @ LREC 2026</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Muzi</namePart>
            <namePart type="family">Matfunjwa</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mmasibidi</namePart>
            <namePart type="family">Setaka</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Rooweither</namePart>
            <namePart type="family">Mabuya</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Menno</namePart>
            <namePart type="family">van Zaanen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>ELRA Language Resources Association (ELRA)</publisher>
            <place>
                <placeTerm type="text">Palma, Mallorca (Spain)</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Models that can estimate the readability of a given text automatically are a valuable resource for any language. There are however many languages for which such models do not work well or simply do not exist yet. In this paper, we lay the groundwork for developing a high-quality application for Afrikaans by having encoder-only language models (LMs) complete a set of cloze tests already completed by humans. Strong correlation between the cloze-test performance of humans and an LM is an indication that the LM could possibly serve as a proxy for human participants. We show that the output of models trained on (some) Afrikaans correlates reasonably well with human answers, underscoring the potential of LMs to be used in automatic readability assessment. A more fine-grained analysis confirms that the correlation is not driven by only a few strongly correlating word classes, but spread relatively evenly over all word classes. We further establish by means of a manual evaluation that, in cases where the cloze-test performance of humans and an LM correlate strongly because both were wrong, LM answers tend to be further off than human answers for the same cloze items. It is noteworthy that the model with the best correlation, afRoBERTa (r=0.62; Spearman’s \ensuremathρ=0.62), is neither the most accurate nor the largest model, but a model trained on Afrikaans only, showing the benefit of small, monolingual LMs compared to large, multilingual models for specific purposes.</abstract>
    <identifier type="citekey">lotz-etal-2026-getting</identifier>
    <identifier type="doi">10.63317/4th5sqwn9wjb</identifier>
    <location>
        <url>https://aclanthology.org/2026.rail-1.11/</url>
    </location>
    <part>
        <date>2026-05</date>
        <extent unit="page">
            <start>107</start>
            <end>120</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Getting Close to Cloze: Investigating Language Model and Human Cloze-test Performance in Afrikaans
%A Lotz, Susan
%A van Noord, Rik
%A van Noord, Gertjan
%Y Matfunjwa, Muzi
%Y Setaka, Mmasibidi
%Y Mabuya, Rooweither
%Y van Zaanen, Menno
%S Proceedings of Resources for African Indigenous Languages (RAIL) 2026 @ LREC 2026
%D 2026
%8 May
%I ELRA Language Resources Association (ELRA)
%C Palma, Mallorca (Spain)
%F lotz-etal-2026-getting
%X Models that can estimate the readability of a given text automatically are a valuable resource for any language. There are however many languages for which such models do not work well or simply do not exist yet. In this paper, we lay the groundwork for developing a high-quality application for Afrikaans by having encoder-only language models (LMs) complete a set of cloze tests already completed by humans. Strong correlation between the cloze-test performance of humans and an LM is an indication that the LM could possibly serve as a proxy for human participants. We show that the output of models trained on (some) Afrikaans correlates reasonably well with human answers, underscoring the potential of LMs to be used in automatic readability assessment. A more fine-grained analysis confirms that the correlation is not driven by only a few strongly correlating word classes, but spread relatively evenly over all word classes. We further establish by means of a manual evaluation that, in cases where the cloze-test performance of humans and an LM correlate strongly because both were wrong, LM answers tend to be further off than human answers for the same cloze items. It is noteworthy that the model with the best correlation, afRoBERTa (r=0.62; Spearman’s \ensuremathρ=0.62), is neither the most accurate nor the largest model, but a model trained on Afrikaans only, showing the benefit of small, monolingual LMs compared to large, multilingual models for specific purposes.
%R 10.63317/4th5sqwn9wjb
%U https://aclanthology.org/2026.rail-1.11/
%U https://doi.org/10.63317/4th5sqwn9wjb
%P 107-120

Markdown (Informal)

[Getting Close to Cloze: Investigating Language Model and Human Cloze-test Performance in Afrikaans](https://aclanthology.org/2026.rail-1.11/) (Lotz et al., RAIL 2026)

• Getting Close to Cloze: Investigating Language Model and Human Cloze-test Performance in Afrikaans (Lotz et al., RAIL 2026)

ACL

• Susan Lotz, Rik van Noord, and Gertjan van Noord. 2026. Getting Close to Cloze: Investigating Language Model and Human Cloze-test Performance in Afrikaans. In Proceedings of Resources for African Indigenous Languages (RAIL) 2026 @ LREC 2026, pages 107–120, Palma, Mallorca (Spain). ELRA Language Resources Association (ELRA).