Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol: studies report non-overlapping metric subsets, preventing direct comparisons. While prior toolkits target subsets of metrics or simultaneous and text-only translation (e.g., stopes-eval, SimulEval, IWSLT), we introduce COMPASS, the first offline S2ST benchmarking framework to unify 46 metrics across eight dimensions with domain-aware validation, and deploy it on 1,248 model-language configurations from FLEURS and CVSS, spanning cascaded and end-to-end architectures over ten language pairs. Architectures exhibit complementary strengths: best-vs-worst gaps exceed 30% on naturalness and speaker preservation but remain within a few points on translation quality, so single-metric rankings systematically misrepresent system quality. Correlation filtering reduces 46 metrics to 10 per direction, with three axes requiring different metrics across X→EN and EN→X (e.g., TER/UTMOS vs. ChrF++/NISQA-MOS); these subsets preserve rankings (Spearman’s 𝜌>0.80) while cutting evaluation time by ≈ 2.5×. Human validation across dubbing, podcasts, and medical domains shows standalone MOS predictors fail to predict listener preference, while top domain-specific metrics correlate with human judgment (𝜌 ≥ 0.90). We release COMPASS as a foundation for domain-aware S2ST evaluation.