<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="bhargava-2020-adaptive">
    <titleInfo>
        <title>Adaptive Transformers for Learning Multimodal Representations</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Prajjwal</namePart>
        <namePart type="family">Bhargava</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Shruti</namePart>
            <namePart type="family">Rijhwani</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jiangming</namePart>
            <namePart type="family">Liu</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yizhong</namePart>
            <namePart type="family">Wang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Rotem</namePart>
            <namePart type="family">Dror</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The usage of transformers has grown from learning about language semantics to forming meaningful visiolinguistic representations. These architectures are often over-parametrized, requiring large amounts of computation. In this work, we extend adaptive approaches to learn more about model interpretability and computational efficiency. Specifically, we study attention spans, sparse, and structured dropout methods to help understand how their attention mechanism extends for vision and language tasks. We further show that these approaches can help us learn more about how the network perceives the complexity of input sequences, sparsity preferences for different modalities, and other related phenomena.</abstract>
    <identifier type="citekey">bhargava-2020-adaptive</identifier>
    <identifier type="doi">10.18653/v1/2020.acl-srw.1</identifier>
    <location>
        <url>https://aclanthology.org/2020.acl-srw.1</url>
    </location>
    <part>
        <date>2020-07</date>
        <extent unit="page">
            <start>1</start>
            <end>7</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Adaptive Transformers for Learning Multimodal Representations
%A Bhargava, Prajjwal
%Y Rijhwani, Shruti
%Y Liu, Jiangming
%Y Wang, Yizhong
%Y Dror, Rotem
%S Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop
%D 2020
%8 July
%I Association for Computational Linguistics
%C Online
%F bhargava-2020-adaptive
%X The usage of transformers has grown from learning about language semantics to forming meaningful visiolinguistic representations. These architectures are often over-parametrized, requiring large amounts of computation. In this work, we extend adaptive approaches to learn more about model interpretability and computational efficiency. Specifically, we study attention spans, sparse, and structured dropout methods to help understand how their attention mechanism extends for vision and language tasks. We further show that these approaches can help us learn more about how the network perceives the complexity of input sequences, sparsity preferences for different modalities, and other related phenomena.
%R 10.18653/v1/2020.acl-srw.1
%U https://aclanthology.org/2020.acl-srw.1
%U https://doi.org/10.18653/v1/2020.acl-srw.1
%P 1-7

Markdown (Informal)

[Adaptive Transformers for Learning Multimodal Representations](https://aclanthology.org/2020.acl-srw.1) (Bhargava, ACL 2020)

• Adaptive Transformers for Learning Multimodal Representations (Bhargava, ACL 2020)

ACL

• Prajjwal Bhargava. 2020. Adaptive Transformers for Learning Multimodal Representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 1–7, Online. Association for Computational Linguistics.