Foundation models have been widely adopted as starting points for fine-tuning, leading to expert-level performance on various downstream tasks. They have been actively developed by improving the pretraining data and/or model architectures. As a result, whenever a new foundation model is released, we have to fine-tune it again on the downstream tasks, requiring non-negligible computational costs as well as data-management costs. A promising approach to this problem is inference-time model update, which modifies the prediction of the new pretrained model using those of the old pretrained/fine-tuned models. However, the existing methods have no theoretical guarantee of improvement in accuracy and may cause unexpected failure. In this paper, we establish the first method for inference-time model update with a theoretical guarantee of improvement in expected rewards, and show that our method can improve performance even when the existing methods fail theoretically. Finally, we empirically validate the effectiveness of our method in practical experiments.

This paper presents DueT, a novel transfer learning method for vision and language models built by contrastive learning. In DueT, adapters are inserted into the image and text encoders, which have been initialized using models pre-trained on uni-modal corpora and then frozen. By training only these adapters, DueT enables efficient learning with a reduced number of trainable parameters. Moreover, unlike traditional adapters, those in DueT are equipped with a gating mechanism, enabling effective transfer and connection of knowledge acquired from pre-trained uni-modal encoders while preventing catastrophic forgetting. We report that DueT outperformed simple fine-tuning, the conventional method fixing only the image encoder and training only the text encoder, and the LoRA-based adapter method in accuracy and parameter efficiency for 0-shot image and text retrieval in both English and Japanese domains.

We tackle the tasks of image and text retrieval using a dual-encoder model in which images and text are encoded independently. This model has attracted attention as an approach that enables efficient offline inferences by connecting both vision and language in the same semantic space; however, whether an image encoder as part of a dual-encoder model can interpret scene-text (i.e., the textual information in images) is unclear. We propose pre-training methods that encourage a joint understanding of the scene-text and surrounding visual information. The experimental results demonstrate that our methods improve the retrieval performances of the dual-encoder models.