Multimodal Large Language Models (MLLMs) have achieved remarkable progress in vision-language interaction, yet their safety alignment remains vulnerable to jailbreak attacks. A key challenge is that safety behavior learned in the textual space does not reliably transfer to fused cross-modal representations, leaving multimodal inputs exploitable through latent semantic cues. We propose __Text-Anchored Semantic Perturbation Attack__ (TA-SPA), a black-box jailbreak framework that optimizes transferable perturbations in a text-anchored semantic space. TA-SPA integrates __Text-Anchored Semantic Factorization__ (TASF), which disentangles cross-modal semantic factors from modality-specific residuals, with __Semantic-Preserving Augmentation__ (SPA), which diversifies harmful target anchors while preserving semantic consistency. Extensive experiments show that TA-SPA consistently outperforms prior jailbreak attacks, transfers to commercial MLLMs, and remains effective under representative defenses, underscoring the need for representation-level safety alignment beyond input-level filtering.