We investigate representational alignment between nine independently trained speech encoders and ten text encoders. We build a controlled corpus in which the same sentences are rendered by different speakers and under different emotional conditions, so that prosody varies while linguistic content is held fixed. We compare every speech–text layer pair and calibrate the strongest match with a permutation procedure that prevents the search over pairs from inflating alignment scores. Local neighborhood agreement between modalities survives calibration throughout and is strongest between late speech layers and early text layers, providing strong evidence of genuine local alignment. Global similarity measures, however, do not provide reliable evidence that the two modalities share a common geometric space as a whole. Alignment is unaffected by speaker identity and prosodic variation, so the shared structure is tied to the content of the sentence itself rather than to how it is spoken. It also does not increase with model capability or with how well a text model reproduces human judgments of meaning, suggesting that this shared structure corresponds more closely to lexical and surface-level representations than to fine-grained sentence meaning. These findings indicate where speech and text representations can be connected, suggest that cross-modal transfer may benefit from exploiting neighborhood structure rather than assuming a shared global geometry, and provide new alignment methods with an existing lexical correspondence to build on, while pointing to sentence-level meaning as the structure that future speech representations could capture so that alignment extends beyond the lexical level.