Reducing the modality gap between image and text representations in CLIP is widely expected to improve cross-modal alignment and downstream performance. However, a smaller average image–text gap does not necessarily lead to consistent accuracy gains. We analyze this mismatch from the perspective of the decision structure in zero-shot classification, i.e. selecting the most similar class-text prototype for an input image. Zero-shot accuracy depends not only on average image–text alignment, but also on class-wise decision margins. Using Linear correction as an analytically tractable case, we show that modality gap correction can alter the relative decision structure among classes and cause predictions to concentrate on a small subset of classes. We refer to this output-space failure mode as prediction-level hubness. Furthermore, experiments across multiple datasets show that accuracy degradation under gap correction is consistently associated with increased prediction concentration, both for Linear correction and for learning-based correction methods. This provides a systematic explanation of why modality gap reduction does not consistently improve CLIP zero-shot accuracy from the perspective of downstream decision structure. Our results suggest that gap correction should be evaluated not only by average alignment, but also by its impact on downstream prediction structure.

We study how a small machine translation model should allocate a fixed data budget between parametric training and retrieval memory. We define a shared 100M-token corpus-access ledger that charges pretraining, fine-tuning, and retrieval-datastore tokens alike. Using ≈42M-parameter models, we compare three uses of the same per-pair datastore across 12 directed pairs of English, Chinese, Japanese, and Korean: leaving it unused, training on it, or retrieving from it into the prompt. We evaluate retrieval additivity across all 12 pairs and analyze retrieval-conditioned behavior in a four-pair core set. Retrieval shows no consistent additive gain over simply not using the datastore, a result that persists under an off-budget dense retriever and pools up to approximately 1000× larger. On the four-pair core set, retrieval-trained models are nevertheless highly context-dependent: removing retrieved examples causes large score drops, while replacing them with query-unrelated, count-matched same-pair examples recovers nearly the full score. This suggests that the learned dependence is primarily on exemplar-shaped context rather than on query-specific information carried by the retrieved neighbors. Our training and evaluation code is available at https://github.com/kheesu/retrieval-without-relevance.

Instruction following is a core capability of modern Large language models (LLMs), making evaluating this capability essential to understanding these models. The Instruction Following Evaluation (IFEval) benchmark from the literature does this using objective criteria, offering a measure of LLM performance without subjective AI or human judgement. However, it only includes English instructions, limiting its ability to assess LLMs in other languages.We propose the Multilingual Instruction Following Evaluation (M-IFEval) benchmark, expanding the evaluation to French, Japanese, and Spanish, with both general and language-specific instructions. Applying this benchmark to 8 state-of-the-art LLMs, we find that benchmark performance across languages and instruction types can vary widely, underscoring the importance of a multilingual benchmark for evaluating LLMs in a diverse cultural context.