High-quality pre-training data is crucial for large language models (LLMs), yet existing score-based selection approaches often fail to balance quality and diversity. Empirical studies have revealed a non-monotonic relationship between dataset scores and downstream performance: simply selecting the top-scored data often leads to performance degradation due to the collapse of correlated evaluation dimensions, which sacrifices diversity. To address this, we propose **O**rthogonal **Di**versity-Aware **S**election (**ODiS**), a novel framework to preserve both quality and diversity by decomposing correlated evaluation metrics into orthogonal feature dimensions. First, ODiS evaluates data from multiple dimensions, covering language quality, knowledge quality, and comprehension difficulty. The resulting multi-dimensional scores are then decorrelated via Principal Component Analysis (PCA), yielding orthogonal evaluation dimensions. For each dimension, a RoBERTa-based scorer is trained to regress the data onto PCA-projected scores, enabling scalable inference on large corpora. Finally, ODiS constructs the training dataset by selecting the top-scored data from each orthogonal dimension, thereby ensuring both quality and diversity. Empirical results show that ODiS-selected data exhibit less than 2% inter-dimension overlap, confirming the orthogonality between dimensions. More importantly, models trained on ODiS-selected data significantly outperform baselines across multiple downstream benchmarks, underscoring the necessity of orthogonal, diversity-aware data selection for LLMs.