Recent work suggests that Whisper-style speech encoders exhibit semantic alignment across languages, based on spoken translation retrieval using representational similarity. If so, this would enable Whisper-style encoders as a viable way to obtain acoustic representations with semantic knowledge, enabling downstream semantic tasks with speech input. However, prior work does not control for phonetic overlap between equivalent utterances, which may artificially support retrieval. We therefore conduct pronunciation-controlled experiments to test whether cross-lingual alignment arises from semantic rather than phonetic similarity, and further probe semantic information in the encoder by way of lexical semantic analogy tasks. Results show that spoken translation retrieval remains strongly above chance with substantially reduced phonetic cues in the final layers of Whisper’s encoder. Furthermore, for both word-level and utterance-level input, Whisper’s encoder representations exhibit non-trivial analogy performance, attaining as high as 25.98% accuracy in an analogy test set. Motivated by this layerwise semantic specialization, we test whether earlier encoder layers preserve representations better suited to ASR in unsupported low-resource languages. Our results confirm it to enable performance gains in automatic speech recognition on low-resource languages unseen during training.