Understanding the geometric structure of pre-trained language model embeddings matters for interpretability and safety. We ask whether sentence-level classification signal lives in the Riemannian geometry of contextual token embeddings, and probe it by extracting per-token pullback metrics from a learned encoder’s analytical Jacobian and aggregating them with the Frechet mean on the symmetric positive definite (SPD) manifold, we call this procedure Riemannian Mean Pooling (RMP). Across three datasets with non-trivial linguistic structure (CoLA, CREAK, RTE), RMP outperforms Euclidean mean pooling, while on FEVER-Symmetric, a benchmark constructed to remove annotation-driven lexical artifacts, the method correctly stays at chance. Ablations show that a randomly initialised encoder combined with Frechet aggregation already beats Euclidean pooling before any training, localising the source of the gain to the geometric aggregation rather than to learned manifold structure. The trained encoder adds a further 1.7 to 2.2 AUC points, consistently across the signal-bearing datasets rather than on any one of them.