<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="levchenko-2025-evaluating">
    <titleInfo>
        <title>Evaluating LLMs for Historical Document OCR: A Methodological Framework for Digital Humanities</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Maria</namePart>
        <namePart type="given">A</namePart>
        <namePart type="family">Levchenko</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-09</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the First on Natural Language Processing and Language Models for Digital Humanities</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Isuri</namePart>
            <namePart type="given">Nanomi</namePart>
            <namePart type="family">Arachchige</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Francesca</namePart>
            <namePart type="family">Frontini</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ruslan</namePart>
            <namePart type="family">Mitkov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Paul</namePart>
            <namePart type="family">Rayson</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>INCOMA Ltd., Shoumen, Bulgaria</publisher>
            <place>
                <placeTerm type="text">Varna, Bulgaria</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Digital humanities scholars increasingly use Large Language Models for historical document digitization, yet lack appropriate evaluation frameworks for LLM-based OCR. Traditional metrics fail to capture temporal biases and period-specific errors crucial for historical corpus creation. We present an evaluation methodology for LLM-based historical OCR, addressing contamination risks and systematic biases in diplomatic transcription. Using 18th-century Russian Civil font texts, we introduce novel metrics including Historical Character Preservation Rate (HCPR) and Archaic Insertion Rate (AIR), alongside protocols for contamination control and stability testing. We evaluate 12 multimodal LLMs, finding that Gemini and Qwen models outperform traditional OCR while exhibiting “over-historicization”—inserting archaic characters from incorrect historical periods. Post-OCR correction degrades rather than improves performance. Our methodology provides digital humanities practitioners with guidelines for model selection and quality assessment in historical corpus digitization.</abstract>
    <identifier type="citekey">levchenko-2025-evaluating</identifier>
    <location>
        <url>https://aclanthology.org/2025.lm4dh-1.7/</url>
    </location>
    <part>
        <date>2025-09</date>
        <extent unit="page">
            <start>75</start>
            <end>85</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Evaluating LLMs for Historical Document OCR: A Methodological Framework for Digital Humanities
%A Levchenko, Maria A.
%Y Arachchige, Isuri Nanomi
%Y Frontini, Francesca
%Y Mitkov, Ruslan
%Y Rayson, Paul
%S Proceedings of the First on Natural Language Processing and Language Models for Digital Humanities
%D 2025
%8 September
%I INCOMA Ltd., Shoumen, Bulgaria
%C Varna, Bulgaria
%F levchenko-2025-evaluating
%X Digital humanities scholars increasingly use Large Language Models for historical document digitization, yet lack appropriate evaluation frameworks for LLM-based OCR. Traditional metrics fail to capture temporal biases and period-specific errors crucial for historical corpus creation. We present an evaluation methodology for LLM-based historical OCR, addressing contamination risks and systematic biases in diplomatic transcription. Using 18th-century Russian Civil font texts, we introduce novel metrics including Historical Character Preservation Rate (HCPR) and Archaic Insertion Rate (AIR), alongside protocols for contamination control and stability testing. We evaluate 12 multimodal LLMs, finding that Gemini and Qwen models outperform traditional OCR while exhibiting “over-historicization”—inserting archaic characters from incorrect historical periods. Post-OCR correction degrades rather than improves performance. Our methodology provides digital humanities practitioners with guidelines for model selection and quality assessment in historical corpus digitization.
%U https://aclanthology.org/2025.lm4dh-1.7/
%P 75-85

Markdown (Informal)

[Evaluating LLMs for Historical Document OCR: A Methodological Framework for Digital Humanities](https://aclanthology.org/2025.lm4dh-1.7/) (Levchenko, LM4DH 2025)

• Evaluating LLMs for Historical Document OCR: A Methodological Framework for Digital Humanities (Levchenko, LM4DH 2025)

ACL

• Maria A. Levchenko. 2025. Evaluating LLMs for Historical Document OCR: A Methodological Framework for Digital Humanities. In Proceedings of the First on Natural Language Processing and Language Models for Digital Humanities, pages 75–85, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria.