<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="blaier-etal-2021-caption">
    <titleInfo>
        <title>Caption Enriched Samples for Improving Hateful Memes Detection</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Efrat</namePart>
        <namePart type="family">Blaier</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Itzik</namePart>
        <namePart type="family">Malkiel</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Lior</namePart>
        <namePart type="family">Wolf</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Marie-Francine</namePart>
            <namePart type="family">Moens</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Xuanjing</namePart>
            <namePart type="family">Huang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lucia</namePart>
            <namePart type="family">Specia</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Scott</namePart>
            <namePart type="given">Wen-tau</namePart>
            <namePart type="family">Yih</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online and Punta Cana, Dominican Republic</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The recently introduced hateful meme challenge demonstrates the difficulty of determining whether a meme is hateful or not. Specifically, both unimodal language models and multimodal vision-language models cannot reach the human level of performance. Motivated by the need to model the contrast between the image content and the overlayed text, we suggest applying an off-the-shelf image captioning tool in order to capture the first. We demonstrate that the incorporation of such automatic captions during fine-tuning improves the results for various unimodal and multimodal models. Moreover, in the unimodal case, continuing the pre-training of language models on augmented and original caption pairs, is highly beneficial to the classification accuracy.</abstract>
    <identifier type="citekey">blaier-etal-2021-caption</identifier>
    <identifier type="doi">10.18653/v1/2021.emnlp-main.738</identifier>
    <location>
        <url>https://aclanthology.org/2021.emnlp-main.738</url>
    </location>
    <part>
        <date>2021-11</date>
        <extent unit="page">
            <start>9350</start>
            <end>9358</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Caption Enriched Samples for Improving Hateful Memes Detection
%A Blaier, Efrat
%A Malkiel, Itzik
%A Wolf, Lior
%Y Moens, Marie-Francine
%Y Huang, Xuanjing
%Y Specia, Lucia
%Y Yih, Scott Wen-tau
%S Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
%D 2021
%8 November
%I Association for Computational Linguistics
%C Online and Punta Cana, Dominican Republic
%F blaier-etal-2021-caption
%X The recently introduced hateful meme challenge demonstrates the difficulty of determining whether a meme is hateful or not. Specifically, both unimodal language models and multimodal vision-language models cannot reach the human level of performance. Motivated by the need to model the contrast between the image content and the overlayed text, we suggest applying an off-the-shelf image captioning tool in order to capture the first. We demonstrate that the incorporation of such automatic captions during fine-tuning improves the results for various unimodal and multimodal models. Moreover, in the unimodal case, continuing the pre-training of language models on augmented and original caption pairs, is highly beneficial to the classification accuracy.
%R 10.18653/v1/2021.emnlp-main.738
%U https://aclanthology.org/2021.emnlp-main.738
%U https://doi.org/10.18653/v1/2021.emnlp-main.738
%P 9350-9358

Markdown (Informal)

[Caption Enriched Samples for Improving Hateful Memes Detection](https://aclanthology.org/2021.emnlp-main.738) (Blaier et al., EMNLP 2021)

• Caption Enriched Samples for Improving Hateful Memes Detection (Blaier et al., EMNLP 2021)

ACL

• Efrat Blaier, Itzik Malkiel, and Lior Wolf. 2021. Caption Enriched Samples for Improving Hateful Memes Detection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9350–9358, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.