Cross-lingual speech-text retrieval for Indic languages (retrieving English text using vernacular voice queries) remains an open problem despite recent progress in multilingual speech and text representation learning. We introduce VaaniCLIP and establish a systematic benchmark on BhasaAnuvaad across Telugu, Tamil, and Hindi, demonstrating that lightweight task-specific contrastive alignment achieves strong performance where established baselines fail. Cascade pipelines (Whisper ASR → NLLB translation → LaBSE retrieval) collapse on low-resource Indic languages: Telugu cascade retrieval reaches only 3.1% R@1, roughly 18× below VaaniCLIP, and SeamlessM4T v2 (2.3B parameters) achieves near-random performance zero-shot. By contrast, VaaniCLIP aligns frozen Whisper and LaBSE encoders with partial Whisper fine-tuning (15M trainable parameters, against a 2.3B-parameter generalist model used zero-shot) and achieves R@1 of 86.4 on Hindi, 55.5 on Telugu, and 54.8 on Tamil at a normalized retrieval pool of N = 2,869, capturing 76–89% of the LaBSE text-text upper bound. A frozen Whisper-medium encoder with only 787K trainable parameters reaches 95–99% of per-language partial fine-tuning performance, confirming encoder representation quality as the primary bottleneck. We further show that the joint model transfers zero-shot to Kannada (an unseen Dravidian language), achieving R@1 of 25.1 (~700× above random) with no Kannada training data, providing evidence that VaaniCLIP learns generalizable Dravidian audio-text alignment.