<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="verhoef-etal-2024-kiki">
    <titleInfo>
        <title>What does Kiki look like? Cross-modal associations between speech sounds and visual shapes in vision-and-language models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Tessa</namePart>
        <namePart type="family">Verhoef</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kiana</namePart>
        <namePart type="family">Shahrasbi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Tom</namePart>
        <namePart type="family">Kouwenhoven</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Tatsuki</namePart>
            <namePart type="family">Kuribayashi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Giulia</namePart>
            <namePart type="family">Rambelli</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ece</namePart>
            <namePart type="family">Takmaz</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Philipp</namePart>
            <namePart type="family">Wicke</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yohei</namePart>
            <namePart type="family">Oseki</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Bangkok, Thailand</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Humans have clear cross-modal preferences when matching certain novel words to visual shapes. Evidence suggests that these preferences play a prominent role in our linguistic processing, language learning, and the origins of signal-meaning mappings. With the rise of multimodal models in AI, such as vision-and-language (VLM) models, it becomes increasingly important to uncover the kinds of visio-linguistic associations these models encode and whether they align with human representations. Informed by experiments with humans, we probe and compare four VLMs for a well-known human cross-modal preference, the bouba-kiki effect. We do not find conclusive evidence for this effect but suggest that results may depend on features of the models, such as architecture design, model size, and training details. Our findings inform discussions on the origins of the bouba-kiki effect in human cognition and future developments of VLMs that align well with human cross-modal associations.</abstract>
    <identifier type="citekey">verhoef-etal-2024-kiki</identifier>
    <location>
        <url>https://aclanthology.org/2024.cmcl-1.17</url>
    </location>
    <part>
        <date>2024-08</date>
        <extent unit="page">
            <start>199</start>
            <end>213</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T What does Kiki look like? Cross-modal associations between speech sounds and visual shapes in vision-and-language models
%A Verhoef, Tessa
%A Shahrasbi, Kiana
%A Kouwenhoven, Tom
%Y Kuribayashi, Tatsuki
%Y Rambelli, Giulia
%Y Takmaz, Ece
%Y Wicke, Philipp
%Y Oseki, Yohei
%S Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics
%D 2024
%8 August
%I Association for Computational Linguistics
%C Bangkok, Thailand
%F verhoef-etal-2024-kiki
%X Humans have clear cross-modal preferences when matching certain novel words to visual shapes. Evidence suggests that these preferences play a prominent role in our linguistic processing, language learning, and the origins of signal-meaning mappings. With the rise of multimodal models in AI, such as vision-and-language (VLM) models, it becomes increasingly important to uncover the kinds of visio-linguistic associations these models encode and whether they align with human representations. Informed by experiments with humans, we probe and compare four VLMs for a well-known human cross-modal preference, the bouba-kiki effect. We do not find conclusive evidence for this effect but suggest that results may depend on features of the models, such as architecture design, model size, and training details. Our findings inform discussions on the origins of the bouba-kiki effect in human cognition and future developments of VLMs that align well with human cross-modal associations.
%U https://aclanthology.org/2024.cmcl-1.17
%P 199-213

Markdown (Informal)

[What does Kiki look like? Cross-modal associations between speech sounds and visual shapes in vision-and-language models](https://aclanthology.org/2024.cmcl-1.17) (Verhoef et al., CMCL-WS 2024)

• What does Kiki look like? Cross-modal associations between speech sounds and visual shapes in vision-and-language models (Verhoef et al., CMCL-WS 2024)

ACL

• Tessa Verhoef, Kiana Shahrasbi, and Tom Kouwenhoven. 2024. What does Kiki look like? Cross-modal associations between speech sounds and visual shapes in vision-and-language models. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 199–213, Bangkok, Thailand. Association for Computational Linguistics.