Umang Sharma
Author directory2026
Lightweight Temporal Attention for Video Retrieval over Frozen Image and Text Encoders
Vidit Khazanchi | Umang Sharma | Shabhareesh Komirishetty
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Vidit Khazanchi | Umang Sharma | Shabhareesh Komirishetty
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Large-scale visual search platforms maintain production text-image embedding models trained on behaviorally-mined query-image pairs, where the text encoder is implicitly tuned to organisation-specific search vocabulary. Extending such a system to video retrieval is non-trivial: retraining risks disrupting hard-won query understanding, while naive temporal aggregation discards the sequential structure that distinguishes a door opening from a door closing. We add a lightweight temporal attention pooling head over independently encoded frame embeddings, keeping both encoders frozen and training only the pooling head and positional encodings. We train on a large-scale dataset from a commercial creative platform using bidirectional InfoNCE with a two-stage hard negative mining pipeline. Benchmarking against LoRA fine-tuned Gemma 3 (12B) and SmolVLM2, neither matches our approach despite substantially larger parameter counts. We also introduce an evaluation benchmark with graded relevance judgments annotated by a VLM judge, where each query is mapped to all relevant videos in a large corpus rather than a single ground-truth clip, avoiding protocols that greatly underestimate model quality. Our model improves nDCG@1 by 10.1% over mean-pooling overall and 25% on complex action queries. With Matryoshka Representation Learning, 256-dim embeddings retain 95.6% of full performance, and int8 quantization costs under 1% in retrieval quality.