<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="hsu-etal-2022-xdbert">
    <titleInfo>
        <title>XDBERT: Distilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Chan-Jan</namePart>
        <namePart type="family">Hsu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hung-yi</namePart>
        <namePart type="family">Lee</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yu</namePart>
        <namePart type="family">Tsao</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Smaranda</namePart>
            <namePart type="family">Muresan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Preslav</namePart>
            <namePart type="family">Nakov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Aline</namePart>
            <namePart type="family">Villavicencio</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Dublin, Ireland</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Transformer-based models are widely used in natural language understanding (NLU) tasks, and multimodal transformers have been effective in visual-language tasks. This study explores distilling visual information from pretrained multimodal transformers to pretrained language encoders. Our framework is inspired by cross-modal encoders’ success in visual-language tasks while we alter the learning objective to cater to the language-heavy characteristics of NLU. After training with a small number of extra adapting steps and finetuned, the proposed XDBERT (cross-modal distilled BERT) outperforms pretrained-BERT in general language understanding evaluation (GLUE), situations with adversarial generations (SWAG) benchmarks, and readability benchmarks. We analyze the performance of XDBERT on GLUE to show that the improvement is likely visually grounded.</abstract>
    <identifier type="citekey">hsu-etal-2022-xdbert</identifier>
    <identifier type="doi">10.18653/v1/2022.acl-short.52</identifier>
    <location>
        <url>https://aclanthology.org/2022.acl-short.52</url>
    </location>
    <part>
        <date>2022-05</date>
        <extent unit="page">
            <start>479</start>
            <end>489</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T XDBERT: Distilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding
%A Hsu, Chan-Jan
%A Lee, Hung-yi
%A Tsao, Yu
%Y Muresan, Smaranda
%Y Nakov, Preslav
%Y Villavicencio, Aline
%S Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
%D 2022
%8 May
%I Association for Computational Linguistics
%C Dublin, Ireland
%F hsu-etal-2022-xdbert
%X Transformer-based models are widely used in natural language understanding (NLU) tasks, and multimodal transformers have been effective in visual-language tasks. This study explores distilling visual information from pretrained multimodal transformers to pretrained language encoders. Our framework is inspired by cross-modal encoders’ success in visual-language tasks while we alter the learning objective to cater to the language-heavy characteristics of NLU. After training with a small number of extra adapting steps and finetuned, the proposed XDBERT (cross-modal distilled BERT) outperforms pretrained-BERT in general language understanding evaluation (GLUE), situations with adversarial generations (SWAG) benchmarks, and readability benchmarks. We analyze the performance of XDBERT on GLUE to show that the improvement is likely visually grounded.
%R 10.18653/v1/2022.acl-short.52
%U https://aclanthology.org/2022.acl-short.52
%U https://doi.org/10.18653/v1/2022.acl-short.52
%P 479-489

Markdown (Informal)

[XDBERT: Distilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding](https://aclanthology.org/2022.acl-short.52) (Hsu et al., ACL 2022)

• XDBERT: Distilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding (Hsu et al., ACL 2022)

ACL

• Chan-Jan Hsu, Hung-yi Lee, and Yu Tsao. 2022. XDBERT: Distilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 479–489, Dublin, Ireland. Association for Computational Linguistics.