<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="lee-etal-2020-vilbertscore">
    <titleInfo>
        <title>ViLBERTScore: Evaluating Image Caption Using Vision-and-Language BERT</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Hwanhee</namePart>
        <namePart type="family">Lee</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Seunghyun</namePart>
        <namePart type="family">Yoon</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Franck</namePart>
        <namePart type="family">Dernoncourt</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Doo</namePart>
        <namePart type="given">Soon</namePart>
        <namePart type="family">Kim</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Trung</namePart>
        <namePart type="family">Bui</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kyomin</namePart>
        <namePart type="family">Jung</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Steffen</namePart>
            <namePart type="family">Eger</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yang</namePart>
            <namePart type="family">Gao</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Maxime</namePart>
            <namePart type="family">Peyrard</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Wei</namePart>
            <namePart type="family">Zhao</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Eduard</namePart>
            <namePart type="family">Hovy</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In this paper, we propose an evaluation metric for image captioning systems using both image and text information. Unlike the previous methods that rely on textual representations in evaluating the caption, our approach uses visiolinguistic representations. The proposed method generates image-conditioned embeddings for each token using ViLBERT from both generated and reference texts. Then, these contextual embeddings from each of the two sentence-pair are compared to compute the similarity score. Experimental results on three benchmark datasets show that our method correlates significantly better with human judgments than all existing metrics.</abstract>
    <identifier type="citekey">lee-etal-2020-vilbertscore</identifier>
    <identifier type="doi">10.18653/v1/2020.eval4nlp-1.4</identifier>
    <location>
        <url>https://aclanthology.org/2020.eval4nlp-1.4</url>
    </location>
    <part>
        <date>2020-11</date>
        <extent unit="page">
            <start>34</start>
            <end>39</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T ViLBERTScore: Evaluating Image Caption Using Vision-and-Language BERT
%A Lee, Hwanhee
%A Yoon, Seunghyun
%A Dernoncourt, Franck
%A Kim, Doo Soon
%A Bui, Trung
%A Jung, Kyomin
%Y Eger, Steffen
%Y Gao, Yang
%Y Peyrard, Maxime
%Y Zhao, Wei
%Y Hovy, Eduard
%S Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems
%D 2020
%8 November
%I Association for Computational Linguistics
%C Online
%F lee-etal-2020-vilbertscore
%X In this paper, we propose an evaluation metric for image captioning systems using both image and text information. Unlike the previous methods that rely on textual representations in evaluating the caption, our approach uses visiolinguistic representations. The proposed method generates image-conditioned embeddings for each token using ViLBERT from both generated and reference texts. Then, these contextual embeddings from each of the two sentence-pair are compared to compute the similarity score. Experimental results on three benchmark datasets show that our method correlates significantly better with human judgments than all existing metrics.
%R 10.18653/v1/2020.eval4nlp-1.4
%U https://aclanthology.org/2020.eval4nlp-1.4
%U https://doi.org/10.18653/v1/2020.eval4nlp-1.4
%P 34-39

Markdown (Informal)

[ViLBERTScore: Evaluating Image Caption Using Vision-and-Language BERT](https://aclanthology.org/2020.eval4nlp-1.4) (Lee et al., Eval4NLP 2020)

• ViLBERTScore: Evaluating Image Caption Using Vision-and-Language BERT (Lee et al., Eval4NLP 2020)

ACL

• Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, and Kyomin Jung. 2020. ViLBERTScore: Evaluating Image Caption Using Vision-and-Language BERT. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 34–39, Online. Association for Computational Linguistics.