Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio-LLM Judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive. We deconstruct the under-specified "naturalness" into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters (Krippendorff’s up to 0.821). Results from benchmarking four MOS predictors and four Audio-LLM Judges reveals that MOS predictors collapse onto acoustic signal quality, while Audio-LLM Judges show selective, prompt-dependent detection that doesnt generalise across the dimension set. Neither class reliably captures linguistically structured speech errors. Our dataset, annotation schema, and evaluation code are publicly released to support more targeted and interpretable TTS evaluation.