<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="stengel-eskin-van-durme-2023-calibrated">
    <titleInfo>
        <title>Calibrated Interpretation: Confidence Estimation in Semantic Parsing</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Elias</namePart>
        <namePart type="family">Stengel-Eskin</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Benjamin</namePart>
        <namePart type="family">Van Durme</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Sequence generation models are increasingly being used to translate natural language into programs, i.e., to perform executable semantic parsing. The fact that semantic parsing aims to predict programs that can lead to executed actions in the real world motivates developing safe systems. This in turn makes measuring calibration—a central component to safety—particularly important. We investigate the calibration of popular generation models across four popular semantic parsing datasets, finding that it varies across models and datasets. We then analyze factors associated with calibration error and release new confidence-based challenge splits of two parsing datasets. To facilitate the inclusion of calibration in semantic parsing evaluations, we release a library for computing calibration metrics.1</abstract>
    <identifier type="citekey">stengel-eskin-van-durme-2023-calibrated</identifier>
    <identifier type="doi">10.1162/tacl_a_00598</identifier>
    <location>
        <url>https://aclanthology.org/2023.tacl-1.69</url>
    </location>
    <part>
        <date>2023</date>
        <detail type="volume"><number>11</number></detail>
        <extent unit="page">
            <start>1213</start>
            <end>1231</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Calibrated Interpretation: Confidence Estimation in Semantic Parsing
%A Stengel-Eskin, Elias
%A Van Durme, Benjamin
%J Transactions of the Association for Computational Linguistics
%D 2023
%V 11
%I MIT Press
%C Cambridge, MA
%F stengel-eskin-van-durme-2023-calibrated
%X Sequence generation models are increasingly being used to translate natural language into programs, i.e., to perform executable semantic parsing. The fact that semantic parsing aims to predict programs that can lead to executed actions in the real world motivates developing safe systems. This in turn makes measuring calibration—a central component to safety—particularly important. We investigate the calibration of popular generation models across four popular semantic parsing datasets, finding that it varies across models and datasets. We then analyze factors associated with calibration error and release new confidence-based challenge splits of two parsing datasets. To facilitate the inclusion of calibration in semantic parsing evaluations, we release a library for computing calibration metrics.1
%R 10.1162/tacl_a_00598
%U https://aclanthology.org/2023.tacl-1.69
%U https://doi.org/10.1162/tacl_a_00598
%P 1213-1231

Markdown (Informal)

[Calibrated Interpretation: Confidence Estimation in Semantic Parsing](https://aclanthology.org/2023.tacl-1.69) (Stengel-Eskin & Van Durme, TACL 2023)

• Calibrated Interpretation: Confidence Estimation in Semantic Parsing (Stengel-Eskin & Van Durme, TACL 2023)

ACL

• Elias Stengel-Eskin and Benjamin Van Durme. 2023. Calibrated Interpretation: Confidence Estimation in Semantic Parsing. Transactions of the Association for Computational Linguistics, 11:1213–1231.