<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="srinivasan-etal-2020-multimodal">
    <titleInfo>
        <title>Multimodal Speech Recognition with Unstructured Audio Masking</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Tejas</namePart>
        <namePart type="family">Srinivasan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ramon</namePart>
        <namePart type="family">Sanabria</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Florian</namePart>
        <namePart type="family">Metze</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Desmond</namePart>
        <namePart type="family">Elliott</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the First International Workshop on Natural Language Processing Beyond Text</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Giuseppe</namePart>
            <namePart type="family">Castellucci</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Simone</namePart>
            <namePart type="family">Filice</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Soujanya</namePart>
            <namePart type="family">Poria</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Erik</namePart>
            <namePart type="family">Cambria</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lucia</namePart>
            <namePart type="family">Specia</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Visual context has been shown to be useful for automatic speech recognition (ASR) systems when the speech signal is noisy or corrupted. Previous work, however, has only demonstrated the utility of visual context in an unrealistic setting, where a fixed set of words are systematically masked in the audio. In this paper, we simulate a more realistic masking scenario during model training, called RandWordMask, where the masking can occur for any word segment. Our experiments on the Flickr 8K Audio Captions Corpus show that multimodal ASR can generalize to recover different types of masked words in this unstructured masking setting. Moreover, our analysis shows that our models are capable of attending to the visual signal when the audio signal is corrupted. These results show that multimodal ASR systems can leverage the visual signal in more generalized noisy scenarios.</abstract>
    <identifier type="citekey">srinivasan-etal-2020-multimodal</identifier>
    <identifier type="doi">10.18653/v1/2020.nlpbt-1.2</identifier>
    <location>
        <url>https://aclanthology.org/2020.nlpbt-1.2</url>
    </location>
    <part>
        <date>2020-11</date>
        <extent unit="page">
            <start>11</start>
            <end>18</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Multimodal Speech Recognition with Unstructured Audio Masking
%A Srinivasan, Tejas
%A Sanabria, Ramon
%A Metze, Florian
%A Elliott, Desmond
%Y Castellucci, Giuseppe
%Y Filice, Simone
%Y Poria, Soujanya
%Y Cambria, Erik
%Y Specia, Lucia
%S Proceedings of the First International Workshop on Natural Language Processing Beyond Text
%D 2020
%8 November
%I Association for Computational Linguistics
%C Online
%F srinivasan-etal-2020-multimodal
%X Visual context has been shown to be useful for automatic speech recognition (ASR) systems when the speech signal is noisy or corrupted. Previous work, however, has only demonstrated the utility of visual context in an unrealistic setting, where a fixed set of words are systematically masked in the audio. In this paper, we simulate a more realistic masking scenario during model training, called RandWordMask, where the masking can occur for any word segment. Our experiments on the Flickr 8K Audio Captions Corpus show that multimodal ASR can generalize to recover different types of masked words in this unstructured masking setting. Moreover, our analysis shows that our models are capable of attending to the visual signal when the audio signal is corrupted. These results show that multimodal ASR systems can leverage the visual signal in more generalized noisy scenarios.
%R 10.18653/v1/2020.nlpbt-1.2
%U https://aclanthology.org/2020.nlpbt-1.2
%U https://doi.org/10.18653/v1/2020.nlpbt-1.2
%P 11-18

Markdown (Informal)

[Multimodal Speech Recognition with Unstructured Audio Masking](https://aclanthology.org/2020.nlpbt-1.2) (Srinivasan et al., nlpbt 2020)

• Multimodal Speech Recognition with Unstructured Audio Masking (Srinivasan et al., nlpbt 2020)

ACL

• Tejas Srinivasan, Ramon Sanabria, Florian Metze, and Desmond Elliott. 2020. Multimodal Speech Recognition with Unstructured Audio Masking. In Proceedings of the First International Workshop on Natural Language Processing Beyond Text, pages 11–18, Online. Association for Computational Linguistics.