Apurva Shah
2024
Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data
Zhongtao Liu
|
Parker Riley
|
Daniel Deutsch
|
Alison Lui
|
Mengmeng Niu
|
Apurva Shah
|
Markus Freitag
Proceedings of the Ninth Conference on Machine Translation
Collecting high-quality translations is crucial for the development and evaluation of machine translation systems. However, traditional human-only approaches are costly and slow. This study presents a comprehensive investigation of 11 approaches for acquiring translation data, including human-only, machine-only, and hybrid approaches. Our findings demonstrate that human-machine collaboration can match or even exceed the quality of human-only translations, while being more cost-efficient. Error analysis reveals the complementary strengths between human and machine contributions, highlighting the effectiveness of collaborative methods. Cost analysis further demonstrates the economic benefits of human-machine collaboration methods, with some approaches achieving top-tier quality at around 60% of the cost of traditional methods. We release a publicly available dataset containing nearly 18,000 segments of varying translation quality with corresponding human ratings to facilitate future research.
Search
Co-authors
- Zhongtao Liu 1
- Parker Riley 1
- Daniel Deutsch 1
- Alison Lui 1
- Mengmeng Niu 1
- show all...
Venues
- wmt1