<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="xu-zhang-2023-triple">
    <titleInfo>
        <title>Triple-Hybrid Energy-based Model Makes Better Calibrated Natural Language Understanding Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Haotian</namePart>
        <namePart type="family">Xu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yingying</namePart>
        <namePart type="family">Zhang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Andreas</namePart>
            <namePart type="family">Vlachos</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Isabelle</namePart>
            <namePart type="family">Augenstein</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Dubrovnik, Croatia</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Though pre-trained language models achieve notable success in many applications, it’s usually controversial for over-confident predictions. Specifically, the in-distribution (ID) miscalibration and out-of-distribution (OOD) detection are main concerns. Recently, some works based on energy-based models (EBM) have shown great improvements on both ID calibration and OOD detection for images. However, it’s rarely explored in natural language understanding tasks due to the non-differentiability of text data which makes it more difficult for EBM training. In this paper, we first propose a triple-hybrid EBM which combines the benefits of classifier, conditional generative model and marginal generative model altogether. Furthermore, we leverage contrastive learning to approximately train the proposed model, which circumvents the non-differentiability issue of text data. Extensive experiments have been done on GLUE and six other multiclass datasets in various domains. Our model outperforms previous methods in terms of ID calibration and OOD detection by a large margin while maintaining competitive accuracy.</abstract>
    <identifier type="citekey">xu-zhang-2023-triple</identifier>
    <identifier type="doi">10.18653/v1/2023.eacl-main.21</identifier>
    <location>
        <url>https://aclanthology.org/2023.eacl-main.21</url>
    </location>
    <part>
        <date>2023-05</date>
        <extent unit="page">
            <start>274</start>
            <end>285</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Triple-Hybrid Energy-based Model Makes Better Calibrated Natural Language Understanding Models
%A Xu, Haotian
%A Zhang, Yingying
%Y Vlachos, Andreas
%Y Augenstein, Isabelle
%S Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
%D 2023
%8 May
%I Association for Computational Linguistics
%C Dubrovnik, Croatia
%F xu-zhang-2023-triple
%X Though pre-trained language models achieve notable success in many applications, it’s usually controversial for over-confident predictions. Specifically, the in-distribution (ID) miscalibration and out-of-distribution (OOD) detection are main concerns. Recently, some works based on energy-based models (EBM) have shown great improvements on both ID calibration and OOD detection for images. However, it’s rarely explored in natural language understanding tasks due to the non-differentiability of text data which makes it more difficult for EBM training. In this paper, we first propose a triple-hybrid EBM which combines the benefits of classifier, conditional generative model and marginal generative model altogether. Furthermore, we leverage contrastive learning to approximately train the proposed model, which circumvents the non-differentiability issue of text data. Extensive experiments have been done on GLUE and six other multiclass datasets in various domains. Our model outperforms previous methods in terms of ID calibration and OOD detection by a large margin while maintaining competitive accuracy.
%R 10.18653/v1/2023.eacl-main.21
%U https://aclanthology.org/2023.eacl-main.21
%U https://doi.org/10.18653/v1/2023.eacl-main.21
%P 274-285

Markdown (Informal)

[Triple-Hybrid Energy-based Model Makes Better Calibrated Natural Language Understanding Models](https://aclanthology.org/2023.eacl-main.21) (Xu & Zhang, EACL 2023)

• Triple-Hybrid Energy-based Model Makes Better Calibrated Natural Language Understanding Models (Xu & Zhang, EACL 2023)

ACL

• Haotian Xu and Yingying Zhang. 2023. Triple-Hybrid Energy-based Model Makes Better Calibrated Natural Language Understanding Models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 274–285, Dubrovnik, Croatia. Association for Computational Linguistics.