<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="robbins-etal-2021-towards">
    <titleInfo>
        <title>Towards Multimodal Vision-Language Models Generating Non-Generic Text</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Wes</namePart>
        <namePart type="family">Robbins</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Zanyar</namePart>
        <namePart type="family">Zohourianshahzadi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jugal</namePart>
        <namePart type="family">Kalita</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 18th International Conference on Natural Language Processing (ICON)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Sivaji</namePart>
            <namePart type="family">Bandyopadhyay</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sobha</namePart>
            <namePart type="given">Lalitha</namePart>
            <namePart type="family">Devi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Pushpak</namePart>
            <namePart type="family">Bhattacharyya</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>NLP Association of India (NLPAI)</publisher>
            <place>
                <placeTerm type="text">National Institute of Technology Silchar, Silchar, India</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Vision-language models can assess visual context in an image and generate descriptive text. While the generated text may be accurate and syntactically correct, it is often overly general. To address this, recent work has used optical character recognition to supplement visual information with text extracted from an image. In this work, we contend that vision-language models can benefit from information that can be extracted from an image, but are not used by current models. We modify previous multimodal frameworks to accept relevant information from any number of auxiliary classifiers. In particular, we focus on person names as an additional set of tokens and create a novel image-caption dataset to facilitate captioning with person names. The dataset, Politicians and Athletes in Captions (PAC), consists of captioned images of well-known people in context. By fine-tuning pretrained models with this dataset, we demonstrate a model that can naturally integrate facial recognition tokens into generated text by training on limited data. For the PAC dataset, we provide a discussion on collection and baseline benchmark scores.</abstract>
    <identifier type="citekey">robbins-etal-2021-towards</identifier>
    <location>
        <url>https://aclanthology.org/2021.icon-main.27</url>
    </location>
    <part>
        <date>2021-12</date>
        <extent unit="page">
            <start>220</start>
            <end>230</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Towards Multimodal Vision-Language Models Generating Non-Generic Text
%A Robbins, Wes
%A Zohourianshahzadi, Zanyar
%A Kalita, Jugal
%Y Bandyopadhyay, Sivaji
%Y Devi, Sobha Lalitha
%Y Bhattacharyya, Pushpak
%S Proceedings of the 18th International Conference on Natural Language Processing (ICON)
%D 2021
%8 December
%I NLP Association of India (NLPAI)
%C National Institute of Technology Silchar, Silchar, India
%F robbins-etal-2021-towards
%X Vision-language models can assess visual context in an image and generate descriptive text. While the generated text may be accurate and syntactically correct, it is often overly general. To address this, recent work has used optical character recognition to supplement visual information with text extracted from an image. In this work, we contend that vision-language models can benefit from information that can be extracted from an image, but are not used by current models. We modify previous multimodal frameworks to accept relevant information from any number of auxiliary classifiers. In particular, we focus on person names as an additional set of tokens and create a novel image-caption dataset to facilitate captioning with person names. The dataset, Politicians and Athletes in Captions (PAC), consists of captioned images of well-known people in context. By fine-tuning pretrained models with this dataset, we demonstrate a model that can naturally integrate facial recognition tokens into generated text by training on limited data. For the PAC dataset, we provide a discussion on collection and baseline benchmark scores.
%U https://aclanthology.org/2021.icon-main.27
%P 220-230

Markdown (Informal)

[Towards Multimodal Vision-Language Models Generating Non-Generic Text](https://aclanthology.org/2021.icon-main.27) (Robbins et al., ICON 2021)

• Towards Multimodal Vision-Language Models Generating Non-Generic Text (Robbins et al., ICON 2021)

ACL

• Wes Robbins, Zanyar Zohourianshahzadi, and Jugal Kalita. 2021. Towards Multimodal Vision-Language Models Generating Non-Generic Text. In Proceedings of the 18th International Conference on Natural Language Processing (ICON), pages 220–230, National Institute of Technology Silchar, Silchar, India. NLP Association of India (NLPAI).