<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="wang-etal-2024-multimodal">
    <titleInfo>
        <title>A Multimodal Large Language Model “Foresees” Objects Based on Verb Information but Not Gender</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Shuqi</namePart>
        <namePart type="family">Wang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Xufeng</namePart>
        <namePart type="family">Duan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Zhenguang</namePart>
        <namePart type="family">Cai</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 28th Conference on Computational Natural Language Learning</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Libby</namePart>
            <namePart type="family">Barak</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Malihe</namePart>
            <namePart type="family">Alikhani</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Miami, FL, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>This study employs the classical psycholinguistics paradigm, the visual world eye-tracking paradigm (VWP), to explore the predictive capabilities of LLAVA, a multimodal large language model (MLLM), and compare them with human anticipatory gaze behaviors. Specifically, we examine the attention weight distributions of LLAVA when presented with visual displays and English sentences containing verb and gender cues. Our findings reveal that LLAVA, like humans, can predictively attend to objects relevant to verbs, but fails to demonstrate gender-based anticipatory attention. Layer-wise analysis indicates that the middle layers of the model are more related to predictive attention than the early or late layers. This study is pioneering in applying psycholinguistic paradigms to compare the multimodal predictive attention of humans and MLLMs, revealing both similarities and differences between them.</abstract>
    <identifier type="citekey">wang-etal-2024-multimodal</identifier>
    <location>
        <url>https://aclanthology.org/2024.conll-1.32</url>
    </location>
    <part>
        <date>2024-11</date>
        <extent unit="page">
            <start>435</start>
            <end>441</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T A Multimodal Large Language Model “Foresees” Objects Based on Verb Information but Not Gender
%A Wang, Shuqi
%A Duan, Xufeng
%A Cai, Zhenguang
%Y Barak, Libby
%Y Alikhani, Malihe
%S Proceedings of the 28th Conference on Computational Natural Language Learning
%D 2024
%8 November
%I Association for Computational Linguistics
%C Miami, FL, USA
%F wang-etal-2024-multimodal
%X This study employs the classical psycholinguistics paradigm, the visual world eye-tracking paradigm (VWP), to explore the predictive capabilities of LLAVA, a multimodal large language model (MLLM), and compare them with human anticipatory gaze behaviors. Specifically, we examine the attention weight distributions of LLAVA when presented with visual displays and English sentences containing verb and gender cues. Our findings reveal that LLAVA, like humans, can predictively attend to objects relevant to verbs, but fails to demonstrate gender-based anticipatory attention. Layer-wise analysis indicates that the middle layers of the model are more related to predictive attention than the early or late layers. This study is pioneering in applying psycholinguistic paradigms to compare the multimodal predictive attention of humans and MLLMs, revealing both similarities and differences between them.
%U https://aclanthology.org/2024.conll-1.32
%P 435-441

Markdown (Informal)

[A Multimodal Large Language Model “Foresees” Objects Based on Verb Information but Not Gender](https://aclanthology.org/2024.conll-1.32) (Wang et al., CoNLL 2024)

• A Multimodal Large Language Model “Foresees” Objects Based on Verb Information but Not Gender (Wang et al., CoNLL 2024)

ACL

• Shuqi Wang, Xufeng Duan, and Zhenguang Cai. 2024. A Multimodal Large Language Model “Foresees” Objects Based on Verb Information but Not Gender. In Proceedings of the 28th Conference on Computational Natural Language Learning, pages 435–441, Miami, FL, USA. Association for Computational Linguistics.