Speech–text retrieval is a multimodal alignment task in which spoken audio and written transcripts are mapped into a shared representation space. Standard retrieval metrics test whether an audio embedding recovers its transcript from a large candidate pool, but not whether the shared representation preserves small grammatical contrasts inside agglutinated words. We construct an automatic Malayalam benchmark with 12,611 controlled audio–text comparison rows derived from 6,611 held-out audio queries. The benchmark contains three morphosyntactic categories—case, tense, and negation—and a lexical numeral-control category. Alternatives are generated with the mlmorph finitestate analyser and generator and retained after a generate–reanalyse consistency check. A CLIP-style model without explicit hard negatives obtains 37.45% full-pool Recall@1 but only 53.66% pairwise accuracy, close to the 50% pairwise baseline. We diagnose a fixedfirst-candidate loader defect that under-delivers later-generated categories and replace it with per-transcript category-aware, access-time resampling. At the matched epoch count, pairwise accuracy rises from 73.50% to 79.76%, and tense accuracy rises from 52.83% to 77.70%. The complete corrected procedure reaches 89.69% pairwise accuracy after extended training, while full-pool retrieval remains below the original checkpoint. A stratified human review of 797 rows retains or corrects 731 (91.7%); the final checkpoint obtains 74.01% on this deliberately reweighted and partly rewritten subset. These results show that global retrieval, controlled grammatical discrimination, candidate-delivery logic, and human benchmark validation are distinct components of multimodal evaluation.