<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kim-etal-2024-semantic">
    <titleInfo>
        <title>Semantic Token Reweighting for Interpretable and Controllable Text Embeddings in CLIP</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Eunji</namePart>
        <namePart type="family">Kim</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kyuhong</namePart>
        <namePart type="family">Shim</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Simyung</namePart>
        <namePart type="family">Chang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sungroh</namePart>
        <namePart type="family">Yoon</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yaser</namePart>
            <namePart type="family">Al-Onaizan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mohit</namePart>
            <namePart type="family">Bansal</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yun-Nung</namePart>
            <namePart type="family">Chen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Miami, Florida, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>A text encoder within Vision-Language Models (VLMs) like CLIP plays a crucial role in translating textual input into an embedding space shared with images, thereby facilitating the interpretative analysis of vision tasks through natural language. Despite the varying significance of different textual elements within a sentence depending on the context, efforts to account for variation of importance in constructing text embeddings have been lacking. We propose a framework of Semantic Token Reweighting to build Interpretable text embeddings (SToRI), which incorporates controllability as well. SToRI refines the text encoding process in CLIP by differentially weighting semantic elements based on contextual importance, enabling finer control over emphasis responsive to data-driven insights and user preferences. The efficacy of SToRI is demonstrated through comprehensive experiments on few-shot image classification and image retrieval tailored to user preferences.</abstract>
    <identifier type="citekey">kim-etal-2024-semantic</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-emnlp.837</url>
    </location>
    <part>
        <date>2024-11</date>
        <extent unit="page">
            <start>14330</start>
            <end>14345</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Semantic Token Reweighting for Interpretable and Controllable Text Embeddings in CLIP
%A Kim, Eunji
%A Shim, Kyuhong
%A Chang, Simyung
%A Yoon, Sungroh
%Y Al-Onaizan, Yaser
%Y Bansal, Mohit
%Y Chen, Yun-Nung
%S Findings of the Association for Computational Linguistics: EMNLP 2024
%D 2024
%8 November
%I Association for Computational Linguistics
%C Miami, Florida, USA
%F kim-etal-2024-semantic
%X A text encoder within Vision-Language Models (VLMs) like CLIP plays a crucial role in translating textual input into an embedding space shared with images, thereby facilitating the interpretative analysis of vision tasks through natural language. Despite the varying significance of different textual elements within a sentence depending on the context, efforts to account for variation of importance in constructing text embeddings have been lacking. We propose a framework of Semantic Token Reweighting to build Interpretable text embeddings (SToRI), which incorporates controllability as well. SToRI refines the text encoding process in CLIP by differentially weighting semantic elements based on contextual importance, enabling finer control over emphasis responsive to data-driven insights and user preferences. The efficacy of SToRI is demonstrated through comprehensive experiments on few-shot image classification and image retrieval tailored to user preferences.
%U https://aclanthology.org/2024.findings-emnlp.837
%P 14330-14345

Markdown (Informal)

[Semantic Token Reweighting for Interpretable and Controllable Text Embeddings in CLIP](https://aclanthology.org/2024.findings-emnlp.837) (Kim et al., Findings 2024)

• Semantic Token Reweighting for Interpretable and Controllable Text Embeddings in CLIP (Kim et al., Findings 2024)

ACL

• Eunji Kim, Kyuhong Shim, Simyung Chang, and Sungroh Yoon. 2024. Semantic Token Reweighting for Interpretable and Controllable Text Embeddings in CLIP. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14330–14345, Miami, Florida, USA. Association for Computational Linguistics.