<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="gasparin-detommaso-2024-distance">
    <titleInfo>
        <title>Distance-aware Calibration for Pre-trained Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Alberto</namePart>
        <namePart type="family">Gasparin</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Gianluca</namePart>
        <namePart type="family">Detommaso</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yaser</namePart>
            <namePart type="family">Al-Onaizan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mohit</namePart>
            <namePart type="family">Bansal</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yun-Nung</namePart>
            <namePart type="family">Chen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Miami, Florida, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Language Models for text classification often produce overconfident predictions for both in-distribution and out-of-distribution samples, i.e., the model’s output probabilities do not match their accuracy. Prior work showed that simple post-hoc approaches are effective for mitigating this issue, but are not robust in noisy settings, e.g., when the distribution shift is caused by spelling mistakes. In this work, we propose Distance Aware Calibration (DAC), a post-hoc approach that changes the confidence scores of a Language Model leveraging the distance between new samples been evaluated and the in-domain training set. We show that using DAC on top of a Language Model can improve in-domain calibration, robustness to different kind of distribution shift and also the model’s ability to detect out-of-distribution samples. We provide an extensive evaluation on common text classification benchmark for both calibration and out-of-distribution detection tasks.</abstract>
    <identifier type="citekey">gasparin-detommaso-2024-distance</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-emnlp.725</url>
    </location>
    <part>
        <date>2024-11</date>
        <extent unit="page">
            <start>12434</start>
            <end>12447</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Distance-aware Calibration for Pre-trained Language Models
%A Gasparin, Alberto
%A Detommaso, Gianluca
%Y Al-Onaizan, Yaser
%Y Bansal, Mohit
%Y Chen, Yun-Nung
%S Findings of the Association for Computational Linguistics: EMNLP 2024
%D 2024
%8 November
%I Association for Computational Linguistics
%C Miami, Florida, USA
%F gasparin-detommaso-2024-distance
%X Language Models for text classification often produce overconfident predictions for both in-distribution and out-of-distribution samples, i.e., the model’s output probabilities do not match their accuracy. Prior work showed that simple post-hoc approaches are effective for mitigating this issue, but are not robust in noisy settings, e.g., when the distribution shift is caused by spelling mistakes. In this work, we propose Distance Aware Calibration (DAC), a post-hoc approach that changes the confidence scores of a Language Model leveraging the distance between new samples been evaluated and the in-domain training set. We show that using DAC on top of a Language Model can improve in-domain calibration, robustness to different kind of distribution shift and also the model’s ability to detect out-of-distribution samples. We provide an extensive evaluation on common text classification benchmark for both calibration and out-of-distribution detection tasks.
%U https://aclanthology.org/2024.findings-emnlp.725
%P 12434-12447

Markdown (Informal)

[Distance-aware Calibration for Pre-trained Language Models](https://aclanthology.org/2024.findings-emnlp.725) (Gasparin & Detommaso, Findings 2024)

• Distance-aware Calibration for Pre-trained Language Models (Gasparin & Detommaso, Findings 2024)

ACL

• Alberto Gasparin and Gianluca Detommaso. 2024. Distance-aware Calibration for Pre-trained Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12434–12447, Miami, Florida, USA. Association for Computational Linguistics.