Ziyuan Wang
Author directoryOther people with similar names: Ziyuan Wang
Unverified author pages with similar names: Ziyuan Wang
2026
Geometry First: Shaping Sentence Embeddings with Controlled Singular Value Regularization
Senyu Li | Hao Yu | Ziyuan Wang | David Ifeoluwa Adelani
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Senyu Li | Hao Yu | Ziyuan Wang | David Ifeoluwa Adelani
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Contrastive learning is widely used for sentence embedding training, yet it provides limited control over the global geometry of the learned embedding space. Sentence embeddings often exhibit spectral concentration, where a small set of directions dominates the representation while many others remain weakly active. We present singular value regularization (SVR), a batch-level framework for shaping sentence embedding spaces by directly controlling the singular value spectrum during training. SVR enables explicit control over both the spectral floor and the number of regularized directions. Experiments show consistent improvements across encoder backbones, pooling strategies, English evaluation tasks, and multilingual settings. Our analysis further shows that the best regularization range is not full-dimensional flattening, but a controlled active-dimensional region aligned with the naturally active subspace of pretrained representations. Moreover, the strongest results are obtained with a large SVR weight, indicating that spectral shaping can provide a substantial optimization signal alongside contrastive supervision. These findings motivate a geometry-aware perspective on sentence embedding learning, in which semantic supervision and explicit control of the embedding-space geometry play complementary roles.