%0 Conference Proceedings %T Towards Efficient Machine Translation Evaluation by Modelling Annotators %A Mathur, Nitika %A Baldwin, Timothy %A Cohn, Trevor %Y Kim, Sunghwan Mac %Y Zhang, Xiuzhen (Jenny) %S Proceedings of the Australasian Language Technology Association Workshop 2018 %D 2018 %8 December %C Dunedin, New Zealand %F mathur-etal-2018-towards %X Accurate evaluation of translation has long been a difficult, yet important problem. Current evaluations use direct assessment (DA), based on crowd sourcing judgements from a large pool of workers, along with quality control checks, and a robust method for combining redundant judgements. In this paper we show that the quality control mechanism is overly conservative, which increases the time and expense of the evaluation. We propose a model that does not rely on a pre-processing step to filter workers and takes into account varying annotator reliabilities. Our model effectively weights each worker’s scores based on the inferred precision of the worker, and is much more reliable than the mean of either the raw scores or the standardised scores. We also show that DA does not deliver on the promise of longitudinal evaluation, and propose redesigning the structure of the annotation tasks that can solve this problem. %U https://aclanthology.org/U18-1010 %P 77-82