<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="chowdhury-etal-2023-apollo">
    <titleInfo>
        <title>APoLLo : Unified Adapter and Prompt Learning for Vision Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Sanjoy</namePart>
        <namePart type="family">Chowdhury</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sayan</namePart>
        <namePart type="family">Nag</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Dinesh</namePart>
        <namePart type="family">Manocha</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The choice of input text prompt plays a critical role in the performance of Vision-Language Pretrained (VLP) models such as CLIP. We present APoLLo, a unified multi-modal approach that combines Adapter and Prompt learning for Vision-Language models. Our method is designed to substantially improve the generalization capabilities of VLP models when they are fine-tuned in a few-shot setting. We introduce trainable cross-attention-based adapter layers in conjunction with vision and language encoders to strengthen the alignment between the two modalities. We enforce consistency between the respective encoder branches (receiving augmented inputs) to prevent overfitting in downstream tasks. Our method is evaluated on three representative tasks: generalization to novel classes, cross-dataset evaluation, and unseen domain shifts. In practice, APoLLo achieves a relative gain up to 6.03% over MaPLe (SOTA) on novel classes for 10 diverse image recognition datasets.</abstract>
    <identifier type="citekey">chowdhury-etal-2023-apollo</identifier>
    <identifier type="doi">10.18653/v1/2023.emnlp-main.629</identifier>
    <location>
        <url>https://aclanthology.org/2023.emnlp-main.629</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>10173</start>
            <end>10187</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T APoLLo : Unified Adapter and Prompt Learning for Vision Language Models
%A Chowdhury, Sanjoy
%A Nag, Sayan
%A Manocha, Dinesh
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F chowdhury-etal-2023-apollo
%X The choice of input text prompt plays a critical role in the performance of Vision-Language Pretrained (VLP) models such as CLIP. We present APoLLo, a unified multi-modal approach that combines Adapter and Prompt learning for Vision-Language models. Our method is designed to substantially improve the generalization capabilities of VLP models when they are fine-tuned in a few-shot setting. We introduce trainable cross-attention-based adapter layers in conjunction with vision and language encoders to strengthen the alignment between the two modalities. We enforce consistency between the respective encoder branches (receiving augmented inputs) to prevent overfitting in downstream tasks. Our method is evaluated on three representative tasks: generalization to novel classes, cross-dataset evaluation, and unseen domain shifts. In practice, APoLLo achieves a relative gain up to 6.03% over MaPLe (SOTA) on novel classes for 10 diverse image recognition datasets.
%R 10.18653/v1/2023.emnlp-main.629
%U https://aclanthology.org/2023.emnlp-main.629
%U https://doi.org/10.18653/v1/2023.emnlp-main.629
%P 10173-10187

Markdown (Informal)

[APoLLo : Unified Adapter and Prompt Learning for Vision Language Models](https://aclanthology.org/2023.emnlp-main.629) (Chowdhury et al., EMNLP 2023)

• APoLLo : Unified Adapter and Prompt Learning for Vision Language Models (Chowdhury et al., EMNLP 2023)

ACL

• Sanjoy Chowdhury, Sayan Nag, and Dinesh Manocha. 2023. APoLLo : Unified Adapter and Prompt Learning for Vision Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10173–10187, Singapore. Association for Computational Linguistics.