Enrique Torres Sanchez

Author directory

2026

Fidelity metrics such as per-token KL divergence (KLD) against a high-precision reference are often used as low-cost proxies for benchmark quality. We test this practice on 28 Qwen3.6-35B-A3B and 41 Devstral-Small-2-24B quantizations across downstream benchmarks. KLD strongly correlates with benchmark score over the full cohorts (𝜌=-0.72 on Qwen and 𝜌=-0.86 on Devstral, both with p<0.001). However, among quantizations retaining at least 99% of measured BF16 composite quality, KLD provides no dependable ranking signal (𝜌=+0.11, p=0.71, n=14 on Qwen; 𝜌=+0.02, p=0.93, n=16 on Devstral). This collapse persists across measurement variants spanning KLD aggregation, perplexity formulations, top-1 agreement, measurement corpora, and context lengths. At the per-prompt level, KLD provides only weak failure-prediction power and fails as a cross-model router across four benchmarks, selecting the better response on only 40.0-49.1% of disagreement prompts and reducing score relative to the better static model on all four. We trace the collapse to a structural decomposition: within the KLD-defined silent zone, KLD primarily measures the volume of disagreement with the reference (composite 𝜌=+0.94, p<0.001 on Qwen; 𝜌=+0.55, p=0.03 on Devstral), while its relationship to the direction of those disagreements is weak and task-conditional.