<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="takishita-etal-2025-llms">
    <titleInfo>
        <title>LLMs Can Compensate for Deficiencies in Visual Representations</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Sho</namePart>
        <namePart type="family">Takishita</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jay</namePart>
        <namePart type="family">Gala</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Abdelrahman</namePart>
        <namePart type="family">Mohamed</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kentaro</namePart>
        <namePart type="family">Inui</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yova</namePart>
        <namePart type="family">Kementchedjhieva</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2025</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Christos</namePart>
            <namePart type="family">Christodoulopoulos</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tanmoy</namePart>
            <namePart type="family">Chakraborty</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Carolyn</namePart>
            <namePart type="family">Rose</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Violet</namePart>
            <namePart type="family">Peng</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Suzhou, China</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-335-7</identifier>
    </relatedItem>
    <abstract>Many vision-language models (VLMs) that prove very effective at a range of multimodal task, build on CLIP-based vision encoders, which are known to have various limitations. We investigate the hypothesis that the strong language backbone in VLMs compensates for possibly weak visual features by contextualizing or enriching them. Using three CLIP-based VLMs, we perform controlled self-attention ablations on a carefully designed probing task. Our findings show that despite known limitations, CLIP visual representations offer ready-to-read semantic information to the language decoder. However, in scenarios of reduced contextualization in the visual representations, the language decoder can largely compensate for the deficiency and recover performance. This suggests a dynamic division of labor in VLMs and motivates future architectures that offload more visual processing to the language decoder.</abstract>
    <identifier type="citekey">takishita-etal-2025-llms</identifier>
    <location>
        <url>https://aclanthology.org/2025.findings-emnlp.825/</url>
    </location>
    <part>
        <date>2025-11</date>
        <extent unit="page">
            <start>15253</start>
            <end>15272</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T LLMs Can Compensate for Deficiencies in Visual Representations
%A Takishita, Sho
%A Gala, Jay
%A Mohamed, Abdelrahman
%A Inui, Kentaro
%A Kementchedjhieva, Yova
%Y Christodoulopoulos, Christos
%Y Chakraborty, Tanmoy
%Y Rose, Carolyn
%Y Peng, Violet
%S Findings of the Association for Computational Linguistics: EMNLP 2025
%D 2025
%8 November
%I Association for Computational Linguistics
%C Suzhou, China
%@ 979-8-89176-335-7
%F takishita-etal-2025-llms
%X Many vision-language models (VLMs) that prove very effective at a range of multimodal task, build on CLIP-based vision encoders, which are known to have various limitations. We investigate the hypothesis that the strong language backbone in VLMs compensates for possibly weak visual features by contextualizing or enriching them. Using three CLIP-based VLMs, we perform controlled self-attention ablations on a carefully designed probing task. Our findings show that despite known limitations, CLIP visual representations offer ready-to-read semantic information to the language decoder. However, in scenarios of reduced contextualization in the visual representations, the language decoder can largely compensate for the deficiency and recover performance. This suggests a dynamic division of labor in VLMs and motivates future architectures that offload more visual processing to the language decoder.
%U https://aclanthology.org/2025.findings-emnlp.825/
%P 15253-15272

Markdown (Informal)

[LLMs Can Compensate for Deficiencies in Visual Representations](https://aclanthology.org/2025.findings-emnlp.825/) (Takishita et al., Findings 2025)

• LLMs Can Compensate for Deficiencies in Visual Representations (Takishita et al., Findings 2025)

ACL

• Sho Takishita, Jay Gala, Abdelrahman Mohamed, Kentaro Inui, and Yova Kementchedjhieva. 2025. LLMs Can Compensate for Deficiencies in Visual Representations. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 15253–15272, Suzhou, China. Association for Computational Linguistics.