Cross-Modality Prompt Transfer (CMPT) has been explored to address the initialization instability and data scarcity challenges inherent in prompt tuning. Its empirical success is typically interpreted as the transfer of linguistic semantics acquired during language-based prompt pretraining. In this paper, we revisit this interpretation and explore what actually transfers when language-pretrained prompts are adapted to the visual modality. Through controlled empirical analyses, we show that transfer performance does not consistently correlate with source-task linguistic performance. Instead, transferability is better explained by the geometric priors of prompt vectors, particularly stable rank and spatial direction relative to target features. To further validate this interpretation, we directly construct prompts with favorable geometry without relying on language pretraining. The resulting prompts achieve competitive transfer performance against language-pretrained prompts across diverse visual tasks, supporting prompt geometry as a stronger empirical explanation of transferability.