2026

We introduce TurCOMET, the first Turkish-specific augmentation of COMET for evaluating translations between Turkish and English. By extending xCOMET-XL with Turkish morphological features and training it on additional Turkish-English data via an additive bottleneck layer, TurCOMET resolves neural evaluation failures caused by sub-word over-segmentation. To validate our metric, we introduce TurkDoc-MT, a meticulously curated Turkish-English document-level aligned dataset of 1,322 instances spanning seven domains. Meta-evaluation demonstrates that TurCOMET significantly outperforms the off-the-shelf baseline, yielding absolute Kendall’s τ correlation gains of +0.113 (system-level) and +0.024 (segment-level), with the largest gains in the TR→EN direction. All models, datasets, code, and annotations are publicly released for the benefit of Turkish natural language processing and benchmarking any translation system.