Human evaluation of MT is essential but exhibits substantial annotator variability that limits evaluation reliability and super- vised learning. Rather than treating dis- agreement as noise or correcting it through protocol changes, we analyze its structure via learned severity classifiers. Across training regimes defined by base- line model reproducibility, we observe in- ternally coherent but mutually incompati- ble severity mappings: models trained on one regime produce confident predictions within that regime but reduced separability on the other. Margin–correctness analysis shows that instability is not uniformly low confidence; separability depends on align- ment between model-internalized and hu- man annotation regimes. These results indicate that unstable MT evaluation regions arise primarily from competing severity interpretations rather than intrinsic example difficulty. Model– annotator disagreement therefore provides a practical signal for identifying unstable evaluation regions during MT evaluation.