<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="barber-etal-2025-active">
    <titleInfo>
        <title>When Does Active Learning Actually Help? Empirical Insights with Transformer-based Automated Scoring</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Justin</namePart>
        <namePart type="given">O</namePart>
        <namePart type="family">Barber</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Michael</namePart>
        <namePart type="given">P</namePart>
        <namePart type="family">Hemenway</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Edward</namePart>
        <namePart type="family">Wolfe</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Coordinated Session Papers</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Joshua</namePart>
            <namePart type="family">Wilson</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Christopher</namePart>
            <namePart type="family">Ormerod</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Magdalen</namePart>
            <namePart type="family">Beiting Parrish</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>National Council on Measurement in Education (NCME)</publisher>
            <place>
                <placeTerm type="text">Wyndham Grand Pittsburgh, Downtown, Pittsburgh, Pennsylvania, United States</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-218-84230-7</identifier>
    </relatedItem>
    <abstract>Developing automated essay scoring (AES) systems typically demands extensive human annotation, incurring significant costs and requiring considerable time. Active learning (AL) methods aim to alleviate this challenge by strategically selecting the most informative essays for scoring, thereby potentially reducing annotation requirements without compromising model accuracy. This study systematically evaluates four prominent AL strategies—uncertainty sampling, BatchBALD, BADGE, and a novel GenAI-based uncertainty approach—against a random sampling baseline, using DeBERTa-based regression models across multiple assessment prompts exhibiting varying degrees of human scorer agreement. Contrary to initial expectations, we found that AL methods provided modest but meaningful improvements only for prompts characterized by poor scorer reliability (\ensuremath&lt;60% agreement per score point). Notably, extensive hyperparameter optimization alone substantially reduced the annotation budget required to achieve near-optimal scoring performance, even with random sampling. Our findings underscore that while targeted AL methods can be beneficial in contexts of low scorer reliability, rigorous hyperparameter tuning remains a foundational and highly effective strategy for minimizing annotation costs in AES system development.</abstract>
    <identifier type="citekey">barber-etal-2025-active</identifier>
    <location>
        <url>https://aclanthology.org/2025.aimecon-sessions.1/</url>
    </location>
    <part>
        <date>2025-10</date>
        <extent unit="page">
            <start>1</start>
            <end>8</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T When Does Active Learning Actually Help? Empirical Insights with Transformer-based Automated Scoring
%A Barber, Justin O.
%A Hemenway, Michael P.
%A Wolfe, Edward
%Y Wilson, Joshua
%Y Ormerod, Christopher
%Y Beiting Parrish, Magdalen
%S Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Coordinated Session Papers
%D 2025
%8 October
%I National Council on Measurement in Education (NCME)
%C Wyndham Grand Pittsburgh, Downtown, Pittsburgh, Pennsylvania, United States
%@ 979-8-218-84230-7
%F barber-etal-2025-active
%X Developing automated essay scoring (AES) systems typically demands extensive human annotation, incurring significant costs and requiring considerable time. Active learning (AL) methods aim to alleviate this challenge by strategically selecting the most informative essays for scoring, thereby potentially reducing annotation requirements without compromising model accuracy. This study systematically evaluates four prominent AL strategies—uncertainty sampling, BatchBALD, BADGE, and a novel GenAI-based uncertainty approach—against a random sampling baseline, using DeBERTa-based regression models across multiple assessment prompts exhibiting varying degrees of human scorer agreement. Contrary to initial expectations, we found that AL methods provided modest but meaningful improvements only for prompts characterized by poor scorer reliability (\ensuremath<60% agreement per score point). Notably, extensive hyperparameter optimization alone substantially reduced the annotation budget required to achieve near-optimal scoring performance, even with random sampling. Our findings underscore that while targeted AL methods can be beneficial in contexts of low scorer reliability, rigorous hyperparameter tuning remains a foundational and highly effective strategy for minimizing annotation costs in AES system development.
%U https://aclanthology.org/2025.aimecon-sessions.1/
%P 1-8

Markdown (Informal)

[When Does Active Learning Actually Help? Empirical Insights with Transformer-based Automated Scoring](https://aclanthology.org/2025.aimecon-sessions.1/) (Barber et al., AIME-Con 2025)

• When Does Active Learning Actually Help? Empirical Insights with Transformer-based Automated Scoring (Barber et al., AIME-Con 2025)

ACL

• Justin O Barber, Michael P. Hemenway, and Edward Wolfe. 2025. When Does Active Learning Actually Help? Empirical Insights with Transformer-based Automated Scoring. In Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Coordinated Session Papers, pages 1–8, Wyndham Grand Pittsburgh, Downtown, Pittsburgh, Pennsylvania, United States. National Council on Measurement in Education (NCME).