Machine Translation Archive

Index to evaluation measures

Publications since 2010

For other periods go to: publications 2005-2009; publications 2000-2004; publications 1990-1999; publications 1970-1989; publications before 1970

To return to home page click here

Adequacy assessment  see User evaluation

Automatic evaluation see Evaluation measures and metrics

Back translation

(2013) Beata Beigman Klebanov & Michael Flor: Associative texture is lost in translation. Proceedings of the Workshop on Discourse in Machine Translation (DiscoMT), Sofia, Bulgaria, August 9, 2013; pp.27-32. [PDF, 309KB]

(2012) Wei He, Hua Wu, Haifeng Wang, & Ting Liu: Improve SMT quality with automatically extracted paraphrase rules. [ACL 2012] Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, Jeju, Republic of Korea, 8-14 July 2012; pp.979-987. [PDF, 330KB]

(2012) Nitin Madnani, Joel Tetreault, & Martin Chodorow: Exploring grammatical error correction with not-so-crummy machine translation. [ACL 2012] The 7th Workshop on the Innovative Use of NLP for Building Educational Applications, Montréal, Canada, June 3-8, 2012; pp.44-53. [PDF, 227KB]

(2011) Fabio Alves, Adrianna Pagano, & Igor L. da Silva: Modeling (un)packing of meaning in translation: insights from effortful text production. Proceedings of the 8th international NLPSC workshop. Special theme: Human-machine interaction in translation, Copenhagen Business School, 20-21 August 2011; ed.Bernadette Sharp, Michael Zock, Michael Carl, Arnt Lykke Jakobsen (Copenhagen Studies in Language 41), Frederiksberg: Samfundslitteratur, 2011; pp.153-163. [PDF, 1064KB]

(2011) Chang Hu, Philip Resnik, Yakov Kronrod, Vladimir Eidelman, Olivia Buzek, & Benjamin B.Bederson: The value of monolingual crowdsourcing in a real-world translation scenrio: simulation using Haitian Creole emergency SMS messages. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.399-404 . [PDF, 164KB]

(2011) Zhifei Li, Jason Eisner, Ziyuan Wang, Sanjeev Khudanpur, & Brian Roark: Minimum imputed risk: unsupervised discriminative training for machine translation. [EMNLP 2011] Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Edinburgh, Scotland, UK, July 27-31, 2011; pp.920-929. [PDF, 236KB]

(2010) Milam Aiken & Mina Park: The efficacy of round-trip translation for MT evaluation. Translation Journal 14 (1), January 2010; 10pp. [PDF, 149KB]

(2010) Yanli Sun, Sharon O’Brien, Minako O’Hagan, & Fred Hollowood: A novel statistical pre-processing model for rule-based machine translation system.  EAMT 2010: Proceedings of the 14th Annual conference of the European Association for Machine Translation, 27-28 May 2010, Saint-Raphaël, France. Proceedings ed.Viggo Hansen and François Yvon; 8pp. [PDF, 790KB]; presentation: 28 slides [PDF, 580KB]

Community content  see Crowd sourcing

Comprehensibility see Reading and comprehension

Confidence measures (see also Quality estimation)

(2015) Christophe Servan, Ngoc Tien Le, Ngoc Quang Luong, Benjamin Lecouteux, & Laurent Besacier: An open-source toolkit for word-level confidence estimation in machine translation. [IWSLT 2015] Proceedings of the International Workshop on Spoken Language Translation, December 3-4, 2015, Da Nang, Vietnam; pp.196-203. [PDF, 3.2MB]

(2014) Sabine Hunsicker & Alexandru Ceausu: Machine translation quality estimation adapted to the translation workflow. Translating and the Computer 36: proceedings. Asling: International Society for Advancement in Language Technology, 27-28 November 2014; pp.133-136. [PDF, 133KB]

(2014) Ngoc-Quang Luong, Laurent Besacier, & Benjamin Lecouteux: An efficient two-pass decoder for SMT using word confidence estimation. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; pp.117-124. [PDF, 621KB]

(2013) Aswarth Dara, Sandipan Dandapat, Declan Groves, & Josef van Genabith: TMTprime: a recommender system for MT and TM integration. [NAACL-HLT 2013] Proceedings of the NAACL HLT 2013 Demonstration Session, 10-12 June 2013, Atlanta, Georgia; pp.10-13. [PDF, 346KB]

(2013) Ngoc Quang Luong, Benjamin Lecouteux & Laurent Besacier: LIG system for WMT13 QE task: investigating the usefulness of features in word confidence estima­tion for MT. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.386-391. [PDF, 202KB]

(2013) Shachar Mirkin, Sriram Venkatapathy, & Marc Dymetman: Confidence-driven rewriting for improved translation. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.257-264. [PDF, 616KB]

(2013) Shachar Mirkin, Sriram Venkatapathy, Marc Dymetman, & Ioan Calapodescu: SORT: an interactive source-rewriting tool for improved translation. ACL-2013: Proceedings of the 51st Meeting of the Association for Computational Linguistics, System demonstrations, Sofia, Bulgaria, August 4-9 2013; pp.85-90. [PDF, 943KB]

(2012) Luong Ngoc Quang: Integrating lexical, syntactic  and system-based features to improve word confidence estimation in SMT. JEP-TALN-RECITAL 2012: Actes de la conference conjointe, Grenoble, 4 au 8 juin 2012; vol.3, pp.43-56. [PDF, 219KB]

(2012) Wei-Yun Ma & Kathleen McKeown: Phrase-level system combination for machine translation based on target-to-target decoding. AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 10pp. [PDF, 111KB]

(2012) Ivan Vulić & Marie-Francine Moens: Detecting highly confident word translations from comparable corpora without any prior knowledge. [EACL 2012] Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, Avignon, France, April 23-27, 2012; pp. 449-459. [PDF, 339KB]

(2012) Qian Yu, Aurélien Max, & François Yvon: Revisiting sentence alignment algorithms for alignment visualization and evaluation. [BUCC 2012] The 5th Workshop on Building and Using Comparable Corpora: “Language Resources for Machine Translation in Less-Resourced Languages and Domains”,  LREC 2012 Workshop, 26 May 2012, Istanbul, Turkey; pp.10-16. [PDF, 430KB]

(2012) Yong Zhuang, Guillaume Wisniewski, & François Yvon: Non-linear models for confidence estimation. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.157-162. [PDF, 427KB]

(2012) Confident MT: estimating translation quality for improved statistical machine translation. [Project paper at] EAMT 2012: Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, May 28-30 2012, ed. Mauro Cettolo, Marcello Federico, Lucia Specia, Andy Way; p.98. [PDF, 424KB]

(2011) Elefterios Avramidis, Maja Popovic, David Vilar, & Aljoscha Burchardt: Evaluate with confidence  estimation: machine ranking of translation outputs using grammatical features. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.65-70. [PDF, 107KB]

(2011) Nguyen Bach, Fei Huang, & Yaser Al-Onaizan: Goodness: a method for measuring machine translation confidence. ACL-HLT 2011: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, Portland, Oregon, June 19-24, 2011; pp.211-219. [PDF, 976KB]

(2011) Christian Hardmeier: Improving machine translation quality prediction with syntactic tree kernels. [EAMT 2011]: proceedings of the 15th conference of the European Association for Machine Translation, 30-31 May 2011, Leuven, Belgium; eds. Mikel L.Forcada, Heidi Depraetere, Vincent Vandeghinste; pp.233-240. [PDF, 633KB] ; presentation, 18 slides [PDF]

(2011) Yanjun Ma, Yifan He, & Josef van Genabith: From the confidence estimation of machine translation to the integration of MT and translation memory. Tutorial at MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.13-15. [PDF, 75KB]

(2011) Lluís Màrquez & Jesús Giménez: Automatic evaluation in machine translation: towards combined linguistically-motivated measures [abstract]. Machine Translation and Morphologically- rich Languages: Research Workshop of the Israel Science Foundation, University of Haifa, Israel, 24 January, 2011; presentation: 101 slides [PDF of PPT, 1248KB]

 (2011) Daniele Pighin & Lluís Màrquez: Automatic projection of semantic structures: an application of pairwise translation ranking. Proceedings of SSST-5, Fifth Workshop on Syntax, Semantics and Structure in Statistical Translation, ACL HLT 2011, Portland, Oregon, USA, June 2011; pp.1-9. [PDF, 142KB]

(2011) Felipe Sánchez-Martínez: Choosing the best machine translation system to translate a sentence by using only source-language information. [EAMT 2011]: proceedings of the 15th conference of the European Association for Machine Translation, 30-31 May 2011, Leuven, Belgium; eds. Mikel L.Forcada, Heidi Depraetere, Vincent Vandeghinste; pp.97-104. [PDF, 338KB] ; presentation, 32 slides [PDF]

(2011) Lucia Specia: Exploiting objective annotations for measuring translation post-editing effort. [EAMT 2011]: proceedings of the 15th conference of the European Association for Machine Translation, 30-31 May 2011, Leuven, Belgium; eds. Mikel L.Forcada, Heidi Depraetere, Vincent Vandeghinste; pp.73-80. [PDF, 365KB]; presentation: 94 slides [PDF]

(2011) Lucia Specia: Linguistic indicators for quality estimation of machine translation [abstract]. [LIHMT] International Workshop on Using Linguistic Information for Hybrid Machine Translation, 18th November 2011, Universitat Politècnica de Catalunya, Barcelona; p.14. [PDF, 139KB]

(2011) Nadi Tomeh, Marco Turchi, Guillaume Wisinewski, Alexandre Allauzen, & François Yvon: How good are your phrases? Assessing phrase quality with single class classification. IWSLT 2011: Proceedings of the International Workshop on Spoken Language Translation, San Francisco, December 8-9, 2011, ed. Marcello Federico, Mei-Yuh Hwang, Margit Rödder, Sebastian Stüker; pp.261-268. [PDF, 691KB]

(2011) Viet Ha-Thuc & Nicola Cancedda: Confidence-weighted learning of factored discriminitave language models. ACL-HLT 2011: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Short papers, Portland, Oregon, June 19-24, 2011; pp.439-444. [PDF,

(2010) Graeme Blackwood, Adrià de Gispert & William Byrne: Fluency constraints for minimum Bayes-risk decoding of statistical machine translation lattices. Coling 2010: 23rd International Conference on Computational Linguistics. Proceedings of the conference, 23-27 August 2010, Beijing International Convention Center, Beijing, China; pp.71-79. [PDF, 547KB]

(2010) Jesús González-Rubio, Daniel Ortiz-Martínez, & Francisco Casacuberta: Balancing user effort and translation error in interactive machine translation via confidence measures. ACL 2010: the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, July 11-16, 2010: Proceedings of the Conference Short Papers; pp.173-177. [PDF, 97KB]

(2010) Jesús González-Rubio, Daniel Ortiz-Martínez, & Francisco Casacuberta: On the use of confidence measures within an interactive-predictive machine translation system. EAMT 2010: Proceedings of the 14th Annual conference of the European Association for Machine Translation, 27-28 May 2010, Saint-Raphaël, France. Proceedings ed.Viggo Hansen and François Yvon; 8pp. [PDF, 525KB]

(2010) Yifan He, Yanjun Ma, Josef van Genabith, & Andy Way: Bridging SMT and TM with translation recommendation. ACL 2010: the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, July 11-16, 2010: Conference proceedings; pp.622-630. [PDF, 573KB]

(2010) Radu Soricut & Abdessamad Echihabi: TrustRank: inducing trust in automatic translations via ranking. ACL 2010: the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, July 11-16, 2010: Conference proceedings; pp.612-621. [PDF, 222KB]

(2010) Lucia Specia & Jesús Giménez: Combining confidence estimation and reference-based metrics for segment-level MT evaluation. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; 10pp. [PDF, 152KB]

(2010) Deyi Xiong, Min Zhang, & Haizhou Li: Error detection for statistical machine translation using linguistic features. ACL 2010: the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, July 11-16, 2010: Conference proceedings; pp.604-611. [PDF, 520KB]

Edit distance

(2014) Friedel Wolff, Laurette Pretorius, & Paul Buitelaar: Missed opportunities in translation memory matching. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.4401-4406. [PDF, 133KB]

(2013) Maarit Koponen: This translation is not too bad: an analysis of post-editor choices in a machine-translation post-editing task. Proceedings of MT Summit XIV Workshop on Post-editing Technology and Practice (WPTP-2), Nice, France, 2 September 2013; pp. 1-9. [PDF, 148KB]

(2013) Maja Popović, Eleftherios Avramidis, Aljoscha Burchardt, David Vilar, & Hans Uszkoreit: What can we learn about the selection mechanism for post-editing?  Proceedings of MT Summit XIV Workshop on Post-editing Technology and Practice (WPTP-2), Nice, France, 2 September 2013; pp. 11-17. [PDF, 160KB]

(2013) Longyue Wang, Derek F.Wong, Lidia S.Chao, Junwen Xing, Yi Lu, & Isabel Trancoso: Edit distance: a new data selection criterion for domain adaptation in SMT. Proceedings of Recent Advances in Natural  Language Processing, Hissar, Bulgaria, 7-13 September 2013; pp.727-732. [PDF, 231KB]

(2012) Christian Buck: Black box features for the WMT 2012 quality estimation shared task. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.91-95. [PDF, 115KB]

(2012) Maarit Koponen: Comparing human perceptions of post-editing effort with post-editing operations. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.181-190. [PDF, 176KB]

(2012) Johannes Leveling, Debasis Ganguly, Sandipan Dandapat, & Gareth J.F.Jones: Approximate sentence retrieval for scalable and efficient example-based machine translation. Proceedings of COLING 2012: Technical Papers, Mumbai, December 2012; pp.1571-1586. [PDF, 291KB]

 (2012) Mengqiu Wang & Christopher D.Manning: SPEDE: probabilistic edit distance metrics for MT evaluation. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.76-83. [PDF, 583KB]

 (2011) Taylor Berg-Kirkpatrick & Dan Klein: Simple effective decipherment via combinatorial optimization. [EMNLP 2011] Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Edinburgh, Scotland, UK, July 27-31, 2011; pp.313-321. [PDF, 299KB]

(2010) Sergio Penkale, Yanjun Ma, Daniel Galron, & Andy Way: Accuracy-based scoring for phrase-based statistical machine translation. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; 10pp. [PDF, 281KB]

Error detection and correction

(2015) Joke Daems, Sonia Vandepitte, Robert Hartsuker, & Lieve Macken: The impact of machine translation error types on post-editing effort indicators. MT Summit XV, October 30 – November 3, 2015, Miami, Florida, USA. Proceedings of MT Summit XV: Fourth Workshop on Post-editing Technology and Practice (WPTP 4); p.31-45. [PDF, 658KB]

(2015) Rohit Kumar, Sanjika Hewavitharana, Nina Zinovieva, Matthew E.Roy, & Edward Pattison-Gordon: Error-tolerant speech-to-speech translation. MT Summit XV, October 30 – November 3, 2015, Miami, Florida, USA. Proceedings of MT Summit XV: vol.1: MT Researchers’ Track; pp.229-239. [PDF, 1.2MB]

(2014) Eleftherios Avramidis, Aljoscha Burchardt, Sabine Hunsicker, Maja Popović, Cindy Tscherwinka, David Vilar, & Hans Uszkoreit: The TaraXŰ corpus of human-annotated  machine translations. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp. 2679-2682. [PDF, 98KB]

(2014) Mara Chinea Rios, Germán Sanchis-Trilles, Daniel Ortiz-Martínez, & Francisco Casacuberta: Online optimisation of log-linear weights in interactive machine translation.  LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.3556-3559. [PDF, 201KB]

(2014) Eunah Cho, Sarah Fünfer, Sebastian Stüker, & Alex Waibel: A corpus of spontaneous speech in lectures: the KIT lecture corpus for spoken language processing and translation.  LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.1554-1559. [PDF, 146KB]

(2014) Angela Costa, Tiago Luís, & Luísa Coheur: Translation errors from English to Portuguese: an annotated corpus. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.1231-1234. [PDF, 407KB]

(2014) Joke Daems, Lieve Macken, & Sonia Vandepitte: On the origin of errors: a fine-grained analysis of MT and PE errors and their relationship. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.62-68. [PDF, 120KB]

(2014) Irina Galinskaya, Valentin Gusev, Elena Meshcheryakova, & Mariya Shmatova: Measuring the impact of spelling errors on the quality of machine translation. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.2683-2689. [PDF, 448KB]

(2014) Homa B.Hashemi & Rebecca Hwa: A comparison of MT errors and ESL errors.  LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.2696-2700. [PDF, 82KB]

(2014) Arle Lommel, Aljoscha Burchardt, Maja Popović, Kim Harris, Eleftherios Avramidis, & Hans Uszkoreit: Using a new analytic measure for the annotation and analysis of MT errors on real data. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; pp.165-172. [PDF, 1781KB]

(2014) Maja Popović, Arle Lommel, Aljoscha Burchardt, Eleftherios Avramidis, & Hans Uszkoreit: Relations between different types of post-editing operations, cognitive effort and temporal effort. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; pp.191-198. [PDF, 402KB]

(2014) Nicholas Ruiz, & Marcello Federico: Assessing the impact of speech recognition errors on machine translation quality. AMTA 2014: proceedings of the eleventh conference of the Association for Machine Translation   in the Americas, Vancouver, BC, October 22-26; pp.261-274. [PDF, 560KB]

(2014) Michel Simard: Clean data for training statistical MT: the case of MT contamination. AMTA 2014: proceedings of the eleventh conference of the Association for Machine Translation in the Americas, Vancouver, BC, October 22-26; pp.69-82. [PDF, 533KB]

(2014) Guillaume Wisniewski, Natalie Kübler, & François Yvon: A corpus of machine translation errors extracted from translation students exercises. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.3585-3588. [PDF, 328KB]

(2013) Anabela Barreiro, Johanna Monti, Brigitte Orliac, & Fernando Batista: When multiwords go bad in machine translation. [MT Summit XIV] Proceedings of the Workshop on Multi-word Units in Machine Translation and Translation Technology, Nice, September 3, 2013; pp.26-33. [PDF, 219KB]

(2013) Ondřej Bojar, Rudolf Rosa & Aleš Tamchyna: Chimera – three heads for English-to-Czech translation.  WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.92-98. [PDF, 175KB]

(2013) Mauro Cettolo, Nicola Bertoldi, & Marcello Federico: Project adaptation for MT-enhanced computer assisted translation. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.27-34. [PDF, 399KB]

(2013) Joke Daems, Lieve Macken, & Sonia Vandepitte: Quality as the sum of its parts: a two-step approach for the identification of translation problems and translation quality assessment for HT and MT+PE.  Proceedings of MT Summit XIV Workshop on Post-editing Technology and Practice (WPTP-2), Nice, France, 2 September 2013; pp. 63-71. [PDF, 209KB]

(2013) Mark Fishel: Ranking translations using error analysis and quality estimation. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.405-407. [PDF, 98KB]

(2013) Meritxell Gonzàlez, Laura Mascarell, & Lluís Màrquez: tSearch: flexible and fast search over automatic translations for improved quality/error analysis.  ACL-2013: Proceedings of the 51st Meeting of the Association for Computational Linguistics, System demonstrations, Sofia, Bulgaria, August 4-9 2013; pp.181-186. [PDF, 963KB]

(2013) Jesús González-Rubio, Daniel Ortiz-Martínez, José-Miguel Benedí, & Francisco Casacuberta: Interactive machine translation using hierarchical translation models. [EMNLP 2013] Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Washington, USA, 18-21 October 2013; pp.244-254. [PDF, 214KB]

(2013) Prashant Mathur, Mauro Cettolo & Marcello Federico: Online learning approaches in computer assisted translation. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.301-308. [PDF, 274KB]

(2013) Thomas Meyer & Lucie Poláková: Machine translation with many manually labeled discourse connectives. Proceedings of the Workshop on Discourse in Machine Translation (DiscoMT), Sofia, Bulgaria, August 9, 2013; pp.43-50. [PDF, 307KB]

(2013) Behzad Mirzababaei, Heshaam Faili, & Nava Ehsan: Discourse-aware statistical machine translation as a context-sensitive spell checker. Proceedings of Recent Advances in Natural  Language Processing, Hissar, Bulgaria, 7-13 September 2013; pp.475-482. [PDF, 1025KB]

(2013) Jan Niehues & Alex Waibel: An MT error-driven discriminative word lexicon using sentence structure features. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.512-520. [PDF, 189KB]

(2013) Maja Popović, Eleftherios Avramidis, Aljoscha Burchardt, Sabine Hunsicker, Sven Schmeier, Cindy Tscherwinka, David Vilar, & Hans Uszkoreit: Learning from human judgments of machine translation output. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.231-238. [PDF, 328KB]

(2013) Michel Simard & George Foster: PEPr: post-edit propagation using phrase-based statistical machine translation. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.191-198. [PDF, 465KB]

(2013) Guillaume Wisniewski, Anil Kumar Singh, Natalia Segal, & François Yvon: Design and analysis of a large corpus of post-edited translations: quality estimation, failure analysis and variability of post-edition. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.117-124. [PDF, 541KB]

(2013) Zheng Yuan & Mariano Felice: Constrained grammatical error correction using statistical machine translation. Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task, Sofia, Bulgaria, August 8-9 2013; pp.52-61. [PDF, 229KB]

(2013) Junguo Zhu, Muyun Yang, Sheng Li, & Tiejun Zhao: Repairing incorrect translation with examples. International Joint Conference on Natural Language Processing, Nagoya, Japan, 14-18 October 2013; pp.967-971. [PDF, 523KB]

(2012) Jan Berka, Ondřej Bojar, Mark Fishel, Maja Popović, & Daniel Zeman: Automatic MT error analysis: Hjerson helping Addicter.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.2158-2163. [PDF, 580KB]

(2012) Nicola Bertoldi, Mauro Cettolo, Marcello Federico, & Christian Buck: Evaluating the learning curve of domain adaptive statistical machine translation systems. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.433-441. [PDF, 141KB]

(2012) Marine Carpuat & Michel Simard: The trouble with SMT consistency. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.442-449. [PDF, 106KB]

(2012) Tagyoung Chung & Michel Galley: Direct error rate minimization for statistical machine translation. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.468-479. [PDF, 673KB]

(2012) Mark Fishel, Ondřej Bojar, & Maja Popović: Terra: a collection of translation error-annotated corpora.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.7-14. [PDF, 367KB]

(2012) Mark Fishel, Rico Sennrich, Maja Popović, & Ondřej Bojar: TerrorCat: a translation error categorization-based MT quality metric. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.64-70. [PDF, 116KB]

(2012) Lluís Formiga & José A.R.Fonollosa: Dealing with input noise in statistical machine translation. Proceedings of COLING 2012: Posters, Mumbai, December 2012; pp.319-328. [PDF, 129KB]

(2012) Meritxell Gonzàlez, Jesús Giménez & Lluís Màrquez: A graphical interface for MT evaluation and error analysis. [ACL 2012] Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, Jeju, Republic of Korea, 10 July 2012, System Demonstrations; pp.139-144. [PDF, 321KB]

(2012) Benjamin Gottesman: Statistical machine translation for depassivizing German part-of-speech sequences. AMTA-2012: Monolingual machine translation-2012 workshop. Proceedings, San Diego, November 1, 2012; pp.55-60. [PDF, 441KB]

(2012) Jie Jiang, Andy Way, & Rejwanul Haque: Translating user-generated content in the social networking space. AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 9pp. [PDF, 130KB]

(2012) Katrin Kirchhoff, Daniel Capurro, & Anne Turner: Evaluating user preferences in machine translation using conjoint analysis. EAMT 2012: Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, May 28-30 2012, ed. Mauro Cettolo, Marcello Federico, Lucia Specia, Andy Way; pp.119-126. [PDF, 223KB]

(2012) Heinrich Kolkhorst, Kevin Kilgour, Sebastian Stüker, & Alex Waibel: Evaluation of interactive user corrections for lecture transcription. IWSLT-2012: 9th International Workshop on Spoken Language Translation, Hong Kong, December 6th-7th, 2012; pp.217-221. [PDF, 635KB]

(2012) Maarit Koponen: Comparing human perceptions of post-editing effort with post-editing operations. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.181-190. [PDF, 176KB]

(2012) Nitin Madnani, Joel Tetreault, & Martin Chodorow: Exploring grammatical error correction with not-so-crummy machine translation. [ACL 2012] The 7th Workshop on the Innovative Use of NLP for Building Educational Applications, Montréal, Canada, June 3-8, 2012; pp.44-53. [PDF, 227KB]

(2012) Tucker Maney, Linda Sibert, Dennis Perzanowski, Kalyan Gupta, & Astrid Schmidt-Nielsen: Toward determining the comprehensibility of machine translations. NAACL-HLT Workshop on Predicting and Improving Text Readability for target reader populations (PITR 2012), Montréal, Canada, June 7, 2012; pp.1-7. [PDF, 318KB]

(2012) Jacob Mundt, Kristen Parton, & Kathleen McKeown: Learning to automatically post-edit dropped words in MT. AMTA-2012: Workshop on post-editing technology and practice. Proceedings, San Diego, October 28, 2012; 8pp. [PDF, 371KB]

(2012) Kristen Parton, Nizar Habash, Kathleen McKeown, Gonzalo Iglesias, & Adrià de Gispert: Can automatic post-editing make MT more meaningful? EAMT 2012: Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, May 28-30 2012, ed. Mauro Cettolo, Marcello Federico, Lucia Specia, Andy Way; pp.111-118. [PDF, 155KB]

(2012) Daniele Pighin, Lluís Màrquez, & Jonathan May: An analysis (and an annotated corpus) of user responses to machine translation output.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.1131-1135. [PDF, 355KB]

(2012) Maja Popović: Class error rates for evaluation of machine translation output. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.71-75. [PDF, 105KB]

(2012) Marion Potet, Emmanuelle Esperança-Rodier, Laurent Besacier, & Hervé Blanchon: Collection of a large database of French-English SMT output corrections.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.4043-4048. [PDF, 357KB]

(2012) Rohit Prasad, Rohit Kumar, Sankaranarayanan Ananthakrishnan, Wei Chen, Sanjika Hewavitharana, Matthew Roy, Frederick Choi, Aaron Challenner, Enoch Kan, Arvid Neelakantan, & Prem Natarajan: Active error detection and resolution for speech-to-speech translation. IWSLT-2012: 9th International Workshop on Spoken Language Translation, Hong Kong, December 6th-7th, 2012; pp.150-157. [PDF, 1156KB]

(2012) Rudolf Rosa, David Mareček, & Ondřej Dušek: DEPFIX: a system for automatic correction of Czech MT outputs. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.362-368. [PDF, 114KB]

(2012) Rudolf Rosa, Ondřej Dušek, David Mareček, & Martin Popel: Using parallel features in parsing of machine-translated sentences for correction of grammatical errors. SSST-6, Sixth Workshop on Syntax, Semantics and Structure in Statistical Translation, Jeju, Republic of Korea, 12 July 2012; pp.39-48. [PDF, 186KB]

(2012) Sara Stymne, Henrik Danielsson, Sofia Bremin, Hongzhan Hu, Johanna Karlsson, Anna Prytz Lillkull, & Martin Wester: Eye tracking as a tool for machine translation error analysis.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.1121-1126. [PDF, 357KB]

(2012) Sara Stymne & Lars Ahrenberg: On the practice of error analysis for machine translation evaluation.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.1785-1790. [PDF, 317KB]

(2012) Dan Tufiş & Ştefan Daniel Dumitrescu: Cascaded phrase-based statistical machine translation systems. EAMT 2012: Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, May 28-30 2012, ed. Mauro Cettolo, Marcello Federico, Lucia Specia, Andy Way; pp.129-136. [PDF, 252KB]

(2012) Justina Valotkaite & Munshi Asadullah: Error detection for post-editing rule-based machine translation. AMTA-2012: Workshop on post-editing technology and practice. Proceedings, San Diego, October 28, 2012; 9pp. [PDF, 117KB]

(2012) David Vilar: DFKI’s SMT system for WMT 2012. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.382-387. [PDF, 90KB]

(2012) Sandra Weiss & Lars Ahrenberg: Error profiling for evaluation of machine-translated text: a Polish-English case study.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.1764-1770. [PDF, 386KB]

(2012) Yue Zhang & Joakim Nivre: Analyzing the effect of global learning and beam-search on transition-based dependency parsing. Proceedings of COLING 2012: Posters, Mumbai, December 2012; pp.1391-1400. [PDF, 282KB]

(2011) Vicent Alabau, Alberto Sanchis, & Francisco Casacuberta: Improving on-line handwritten recognition using translation models in multimodal interactive machine translation. ACL-HLT 2011: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Short papers, Portland, Oregon, June 19-24, 2011; pp.389-394. [PDF, 417KB]

(2011) Sankaranarayanan Ananthakrishnan, Shiv Vitaladevuni, Rohit Prasad, & Prem Natarajan: Source error-projection for sample selection in phrase-based SMT for resource-poor languages. [IJCNLP 2011] Proceedings of the 5th International Joint Conference on Natural Language Processing, Chiang Mai, Thailand, November 8-13, 2011; pp.819-827. [PDF, 425KB]

(2011) Nguyen Bach, Fei Huang, & Yaser Al-Onaizan: Goodness: a method for measuring machine translation confidence. ACL-HLT 2011: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, Portland, Oregon, June 19-24, 2011; pp.211-219. [PDF, 976KB]

(2011) Ondřej Bojar: Analyzing error types in English-Czech machine translation. Prague Bulletin of Mathematical Linguistics 95, April 2011; pp.63-76. [PDF, 166KB]

(2011) Ahmed El Kholy & Nizar Habash: Automatic error analysis for morphologically rich languages. MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.225-232. [PDF, 219KB]

(2011) Shasha Liao, Cheng Wu, & Juan Huerta: Evaluating human correction quality for machine translation from crowdsourcing.  [RANLP 2011] Proceedings of Recent Advances in Natural Language Processing, Hissar, Bulgaria, 12-14 September 2011; pp.598-603. [PDF, 487KB]

(2011) David Mareček, Rudolf Rosa, Petra Galuščáková, & Ondřej Bojar: Two-step translation with grammatical post-processing. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.426-432. [PDF, 119KB]

(2011) Maja Popović: Hjerson: an open source tool for automatic error classification of machine translation output. Sixth Machine Translation Marathon, 5-10 September 2011, Trento; Prague Bulletin of Mathematical Linguistics, no.96, October 2011; pp.59-67. [PDF, 121KB]; presentation, 20 slides [PDF of PPT, 226KB]

(2011) Maja Popović & Aljoscha Burchardt: From human to automatic error classification for machine translation output. [EAMT 2011]: proceedings of the 15th conference of the European Association for Machine Translation, 30-31 May 2011, Leuven, Belgium; eds. Mikel L.Forcada, Heidi Depraetere, Vincent Vandeghinste; pp.265-272. [PDF, 183KB]; presentation, 16 slides [PDF, 385KB]

(2011) Maja Popović & Hermann Ney: Towards automatic error analysis of machine translation output. Computational Linguistics 37 (4), pp. 657-688. [PDF, 219KB]

(2011) Sara Stymne: BLAST: a tool for error analysis of machine translation output. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Portland, Oregon, June 19-24, 2011; pp.56-61. [PDF, 128KB]

(2011) Sara Stymne: Pre- and postprocessing for statistical machine translation into Germanic languages. ACL-HLT 2011: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Student Session, Portland, Oregon, June 19-24, 2011; pp.12-17. [PDF, 93KB]

(2011) Kaveh Taghipour, Shahram Khadivi, & Jia Xu: Parallel corpus refinement as an outlier detection algorithm. MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.414-421. [PDF, 213KB]

(2011) Dan Zeman, Mark Fishel, Jan Berka, & Ondřej Bojar: Addicter: what’s wrong with my translations? Sixth Machine Translation Marathon, 5-10 September 2011, Trento; Prague Bulletin of Mathematical Linguistics, no.96, October 2011; pp.79-88. [PDF, 276KB]; presentation, 20 slides [PDF of PPT, 359KB]

(2010) Nicola Bertoldi, Mauro Cettolo, & Marcello Federico: Statistical machine translation of texts with misspelled words. NAACL HLT 2010: Human Language Technologies: the 2010 annual conference of the North American Chapter of the Association for Computational Linguistics. Proceedings… June 2-4, 2010, Los Angeles, California; pp.412-419. [PDF, 160KB]

(2010) Graeme Blackwood, Adrià de Gispert & William Byrne: Fluency constraints for minimum Bayes-risk decoding of statistical machine translation lattices. Coling 2010: 23rd International Conference on Computational Linguistics. Proceedings of the conference, 23-27 August 2010, Beijing International Convention Center, Beijing, China; pp.71-79. [PDF, 547KB]

(2010) Mireia Farrús, Marta R.Costa-jussà, José B.Mariño, & José A.R.Fonollosa: Linguistic-based evaluation criteria to identift statistical machine translation errors. EAMT 2010: Proceedings of the 14th Annual conference of the European Association for Machine Translation, 27-28 May 2010, Saint-Raphaël, France. Proceedings ed.Viggo Hansen and François Yvon; 7pp. [PDF, 606KB]

(2010) Maxim Khalilov, Lauma Pretkalniņa, Natalja Kuvaldina, & Veronika Pereseina: SMT of Latvian, Lithuanian and Estonian languages: a comparative study. Human Language Technologies – the Baltic Perpective,  International Conference, Riga, Latvia, October 8, 2010; 8pp. [PDF, 145KB]

(2010) Kristen Parton & Kathleen McKeown: MT error detection for cross-lingual question answering. Coling 2010: 23rd International Conference on Computational Linguistics, 23-27 August 2010, Beijing International Convention Center, Beijing, China, Posters volume; pp.946-954. [PDF, 161KB]

(2010) Susanne Preuß, Christoph Rösener, & Paul Schmidt: Language technology for automatic control in eLearning tools – translator and second language training. Translating and the Computer 32, 18-19 November 2010, London; 10pp. [PDF, 486KB]

(2010) Philip Resnik, Olivia Buzek, Chang Hu, Yakov Kronrod, Alex Quinn, & Benjamin B.Bederson: Improving translation via targeted paraphrasing. [EMNLP 2010] Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, MIT, Massachusetts, USA, 9-11 October 2010; pp.127-137. [PDF, 548KB]

(2010) Sara Stymne & Lars Ahrenberg: Using a grammar checker for evaluation and postprocessing of statistical machine translation. LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.2175-2181. [PDF,

(2010) Guillaume Wisniewski, Alexandre Allauzen, & François Yvon: Assessing phrase-based translation models with oracle decoding. [EMNLP 2010] Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, MIT, Massachusetts, USA, 9-11 October 2010; pp.933-943. [PDF, 188KB]

(2010) Deyi Xiong, Min Zhang, & Haizhou Li: Error detection for statistical machine translation using linguistic features. ACL 2010: the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, July 11-16, 2010: Conference proceedings; pp.604-611. [PDF, 520KB]

(2010) Huichao Xue & Rebecca Hwa: Syntax-driven machine translation as a model of ESL revision. Coling 2010: 23rd International Conference on Computational Linguistics, 23-27 August 2010, Beijing International Convention Center, Beijing, China, Posters volume; pp.1373-1381. [PDF, 258KB]

Evaluation measures and metrics (general)

(2015) Zied Elloumi, Hervé Blanchon, Gilles Serasset, & Laurent Besacier: METEOR for multiple target languages using DBnary. MT Summit XV, October 30 – November 3, 2015, Miami, Florida, USA. Proceedings of MT Summit XV: vol.1: MT Researchers’ Track; pp.80-89. [PDF, 569KB]

(2015) Alan K. Melby: QT21: a new era for translators and the computer. Proceedings of the 37th Conference Translating and the Computer, London, November 26-27, 2015; pp.1-11. [PDF, 535 KB]

(2015) Alan K.Melby: Quality evaluation of four translations of a kidney document: focus on reliability. MT Summit XV, October 30 – November 3, 2015, Miami, Florida, USA. Proceedings of MT Summit XV: vol.2: MT Users’ Track; pp.65-68. [PDF, 547KB]

(2015) Elaine O’Curran: MT quality evaluations: from test environment to production. MT Summit XV, October 30 – November 3, 2015, Miami, Florida, USA. Proceedings of MT Summit XV: vol.2: MT Users’ Track; pp.106-143. [PDF, 1516KB]

(2015) Carla Parra Escartín & Manuel Arcedillo: Machine translation evaluation made fuzzier: a study on post-editing productivity and evaluation metrics in commercial settings. MT Summit XV, October 30 – November 3, 2015, Miami, Florida, USA. Proceedings of MT Summit XV: vol.1: MT Researchers’ Track; pp.131-144. [PDF, 491KB]

(2015) Carla Parra Escartin & Manuel Arcedillo: Living on the edge: productivity gain thresholds in machine translation evaluation metrics. MT Summit XV, October 30 – November 3, 2015, Miami, Florida, USA. Proceedings of MT Summit XV: Fourth Workshop on Post-editing Technology and Practice (WPTP 4); p.46-56. [PDF, 858KB]

(2015) Marianne Starlander: Let the EAGLES fly into new standards: adapting our CAT tool evaluation methodology to the ISO 25000 series. Proceedings of the 37th Conference Translating and the Computer, London, November 26-27, 2015; pp .96-101. [PDF, 336KB]

(2015) Mihaela Vela & Ekaterina Lapshinova-Koltunski: Register-based machine translation evaluation with text classification techniques. MT Summit XV, October 30 – November 3, 2015, Miami, Florida, USA. Proceedings of MT Summit XV: vol.1: MT Researchers’ Track; pp.215-228. [PDF, 506KB]

(2014) Sandra Antunes & Amália Mendes: An evaluation of the role of statistical measures and frequency for MWE identification. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.4046-4051. [PDF, 138KB]

(2014) Eleftherios Avramidis, Aljoscha Burchardt, Sabine Hunsicker, Maja Popović, Cindy Tscherwinka, David Vilar, & Hans Uszkoreit: The TaraXŰ corpus of human-annotated  machine translations. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp. 2679-2682. [PDF, 98KB]

(2014) Petra Barančíková: Parmesan: Meteor without paraphrases with paraphrased references. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.355-361. [PDF, 305KB]

(2014) Ergun Biçici & Andy Way: Referential translation machines for predicting translation quality. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.313-321. [PDF, 378KB]

(2014) Lars Borin, Jens Allwood, & Gerard de Melo: Bring vs.MTRoget: evaluating automatic thesaurus translation. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.2115-2121. [PDF, 252KB]

(2014) Petra Barančiková, Rudolf Rosa, & Aleš Tamchyna: Improving evaluation of English-Czech MT through paraphrasing.  LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.2696-2700. [PDF, 127KB]

(2014) Boxing Chen & Colin Cherry: A systematic comparison of smoothing techniques for sentence-level BLEU. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.362-367. [PDF, 316KB]

(2014) E.Comelles, J.Atserias, V.Arranz, I.Castellón, & J.Sesé: VERTa: facing the multilingual experience of a linguistically-based MT evaluation. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.2701-2707. [PDF, 163KB]

(2014) Elisabet Comelles & Jordi Atserias: VERTa participation in the WMT14 metrics task. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.368-375. [PDF, 255KB]

(2014) Oliver Čulo: Approaching machine translation from translation studies – a perspective on commonalities, potentials, differences. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; pp. 199-206. [PDF, 361KB]

(2014) Michael Denkowski & Alon Lavie: Meteor Universal: language specific translation evaluation for any target language. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.376-380. [PDF, 282KB]

(2014) Hiroshi Echizen’ya, Kenji Araki, & Eduard Hovy: Application of prize based on sentence length in chunk-based automatic evaluation of machine translation. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.381-386. [PDF, 315KB]

(2014) Thierry Etchegoyhen, Lindsay Bywood, Mark Fishel, Panayota Georgakopoulou, Jie Jiang, Gerard van Loenhout, Arantza del Pozo, Mirjam Sepesy Maučec, Anja Turner, & Martin Volk: Machine translation for subtitling: a large-scale evaluation. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.46-53. [PDF, 301KB]

(2014) Shubham Gautam & Pushpak Bhattacharyya: LAYERED: metric for machine translation evaluation. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.387-393. [PDF, 406KB]

(2014) Meritxell Gonzàlez, Alberto Barrón-Cedeño, & Lluís Màrquez: IPA and STOUT: leveraging linguistic and source-based features for machine translation evaluation. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.394-401. [PDF, 319KB]

(2014) Shinsuke Goto, Donghui Lin, & Toru Ishida: Crowdsourcing for evaluating machine translation quality. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.3456-3463. [PDF, 214]

(2014) Yvette Graham, Nikita Mathur, & Timothy Baldwin: Randomized significance tests in machine translation. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.266-274. [PDF, 358KB]

(2014) Teresa Herrmann, Jan Niehues, & Alex Waibel: Manual analysis of structurally informed reordering in German-English machine translation. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.4379-4386. [PDF, 127KB]

(2014) Hideki Isozaki, Natsume Kouchi, & Tsutomu Hirao: Dependency-based automatic enumeration of semantically equivalent word orders for evaluating Japanese translations. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.287-292. [PDF, 264KB]

(2014) Guillaume Jacquet, Maud Ehrmann, & Ralf Steinberger: Clustering of multi-word named entity variants: multilingual evaluation. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.2696-2700. [PDF, 217KB]

(2014) Shafiq Joty, Francisco Guzmán, Lluís Màrquez & Preslav Nakov: DiscoTK: using discourse structure for machine translation evaluation. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.402-408. [PDF, 426KB]

(2014) Adam Kilgarriff, Miloš Jakubíček, Vojtěch Kovář, Pavel Rychlý, Vít Baisa, & Lucia Kocincová: Extrinsic corpus evaluation with a collocation dictionary task. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.545-552. [PDF, 143KB]

(2014) Valia Kordoni & Iliana Simova: Multiword expressions in machine translation. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.1208-1211. [PDF, 110KB]

(2014) Jindřich Libovický & Pavel Pecina: Tolerant BLEU: a submission to the WMT14 metrics task. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.409-413. [PDF, 360KB]

(2014) Chi-kiu Lo & Dekai Wu: On the reliability and inter-annotator agreement of human semantic MT evaluation via HMEANT. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.602-607. [PDF, 279KB]

(2014) Matouš Macháček & Ondřej Bojar: Results of the WMT14 metrics shared task. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.293-301. [PDF, 331KB]

(2014) Linda Mitchell, Johann Roturier, & David Silva: Using the ACCEPT framework to conduct an online community-based translation evaluation study. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; pp.155-158. [PDF, 361KB]

(2014) John Moran, Christian Saam, & Dave Lewis: Towards desktop-based CAT tool instrumentation. AMTA 2014: proceedings of the eleventh conference of the Association for Machine Translation in the Americas, Vancouver, BC, October 22-26; Workshop on Post-editing Technology and Practice (WPTP-3); pp.99-112. [PDF, 772]

(2014) Joaquim Moré & Salvador Climent: Machine translationness: machine-likeness in machine translation evaluation. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.54-61. [PDF, 464KB]

(2014) Sharon O’Brien, Joss Moorkens, & Joris Vreeke: Kanjingo – a mobile app for post-editing. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; pp.137-141. [PDF, 736KB]

(2014) Nicholas Ruiz & Marcello Federico: Complexity of spoken versus written language for machine translation. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; pp.173-180. [PDF, 479KB]

(2014) Keisuke Sakaguchi, Matt Post, & Benjamin Van Durme: Efficient elicitation of annotations for human evaluation of machine translation. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.1-11. [PDF, 641KB]

(2014) Miloš Stanojević & Khalil Sima’an: BEER: Better evaluation as ranking. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.414-419. [PDF, 291KB]

(2014) Sitong Yang, Heng Yu, Hongmei Zhao, Qun Liu, & Yajuan Lü: Review and analysis of China workshop on machine translation 2013 evaluation. AMTA 2014: proceedings of the eleventh conference of the Association for Machine Translation in the Americas, Vancouver, BC, October 22-26; pp.208-221. [PDF, 767KB]

 (2014) QTLaunchPad: preparation and launch of a large-scale action for quality translation technology. Project duration: July 2012 – June 2014. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; p.133. [PDF, 484KB]

(2014) QTLeap: quality translation with deep language engineering approaches. Project duration: November 2013 – October 2016. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; p.129. [PDF, 514KB]

 (2013) Jacob Andreas, Andreas Vlachos, & Stephen Clark: Semantic parsing as machine translation. ACL-2013: Proceedings of the 51st Meeting of the Association for Computational Linguistics, Short papers, Sofia, Bulgaria, August 4-9 2013; pp.47-52. [PDF, 133KB]

(2013) Alberto Barrón-Cedeño, Lluís Màrquez, Maria Fuentes, Horacio Rodríguez, & Jordi Turmo: UPC-CORE: what can machine evaluation metrics and Wikipedia do for estimating semantic textual similarity? [SEM-2013] Second Joint Conference on Lexical and Computational Semantics, Atlanta, Georgia, June 13-14, 2013; vol.1, pp.143-147. [PDF, 103KB]

(2013) Ergun Biçici & Josef van Genabith: CNGL: grading student answers by acts of translation. [SEM-2013] Second Joint Conference on Lexical and Computational Semantics, Volume 2: Seventh International Workshop on Semantic Evaluation, Atlanta, Georgia, June 14-15, 2013; pp.585-591. [PDF, 129KB]

(2013) Ergun Biçici & Josef van Genabith: CNGL-CORE: referential translation machines for measuring semantic similarity. [SEM-2013] Second Joint Conference on Lexical and Computational Semantics, Atlanta, Georgia, June 13-14, 2013; vol.1, pp.234-240. [PDF, 129KB]

(2013) Ergun Bicici: Referential translation machines for quality estimation. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.343-351. [PDF, 300KB]

(2013) Alexandra Birch, Barry Haddow, Ulrich Germann, Maria Nadejde, Christian Buck & Philipp Koehn: The feasibility of HMEANT as a human MT evaluation metric. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.52-61. [PDF, 222KB]

(2013) Marine Carpuat: A semantic evaluation of machine translation lexical choice. [SSST-7]: Proceedings of the Seventh Workshop on Syntax, Semantics and Structure in Statistical Translation. Atlanta, Georgia, 13 June 2013; pp.1-10. [PDF, 145KB]

(2013) Joke Daems, Lieve Macken, & Sonia Vandepitte: Quality as the sum of its parts: a two-step approach for the identification of translation problems and translation quality assessment for HT and MT+PE.  Proceedings of MT Summit XIV Workshop on Post-editing Technology and Practice (WPTP-2), Nice, France, 2 September 2013; pp. 63-71. [PDF, 209KB]

(2013) Melania Duma, Cristina Vertan, & Wolfgang Menzel: A new syntatctic metric for evaluation of machine translation.  ACL Student Research Workshop, Sofia, Bulgaria, August 4-9 2013; pp.130-135. [PDF, 346KB]

(2013) Hiroshi Echizen’ya, Kenji Araki, & Eduard Hovy: Automatic evaluation metric for machine translation that is independent of sentence length. Proceedings of Recent Advances in Natural  Language Processing, Hissar, Bulgaria, 7-13 September 2013; pp.230-236. [PDF, 152KB]

(2013) Meritxell Gonzàlez, Laura Mascarell, & Lluís Màrquez: tSearch: flexible and fast search over automatic translations for improved quality/error analysis.  ACL-2013: Proceedings of the 51st Meeting of the Association for Computational Linguistics, System demonstrations, Sofia, Bulgaria, August 4-9 2013; pp.181-186. [PDF, 963KB]

(2013) Aaron Li-Feng Han, Derek F. Wong, Lidia S. Chao, Yi Lu, Liangye He, Yiming Wang & Jiaji Zhou: A description of tunable machine translation evaluation systems in WMT13 metrics task. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.414-421. [PDF, 330KB]

(2013) Aaron Li-Feng Han, Derek F.Wong, Lidia S.Chao, Liangye He, Yi Li, Junwen Xing, & Xiaodong Zeng: Language-independent model for machine translation evaluation with reinforced factors. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.215-222. [PDF, 623KB]

(2013) Ekaterina Lapshinova-Koltunski: VARTRA: a comparable corpus for analysis of translation variation. Proceedings of the 6th Workshop on Building and Using Comparable Corpora, Sofia, Bulgaria, August 8, 2013; pp.77-86. [PDF, 336KB]

(2013) Haibo Li, Jing Zheng, Heng Ji, Qi Li, & Wen Wang: Name-aware machine translation. ACL-2013: Proceedings of the 51st Meeting of the Association for Computational Linguistics, Sofia, Bulgaria, August 4-9 2013; pp.604-614. [PDF, 443KB]

(2013) Maoxi Li, Aiwen Jiang, & Mingwen Wang: Listwise approach to learning to rank for automatic evaluation of machine translation. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.51-59. [PDF, 277KB]

(2013) Chi-kiu Lo & Dekai Wu: Can informal genres be better translated by tuning on automatic semantic metrics? Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.93-100. [PDF, 598KB]

(2013) Chi-kiu Lo & Dekai Wu: Human semantic MT evaluation with HMEANT for IWSLT 2013. [IWSLT 2013] Proceedings of the 10th International Workshop on Spoken Language Translation, Heidelberg, Germany, Dec.5-6, 2013; 7pp. [PDF, 317KB]

(2013) Chi-kiu Lo, Meriem Beloucif, & Dekai Wu: Improving machine translation into Chinese by tuning against Chinese MEANT. [IWSLT 2013] Proceedings of the 10th International Workshop on Spoken Language Translation, Heidelberg, Germany, Dec.5-6, 2013; 8pp. [PDF, 284KB]

(2013) Chi-kiu Lo, Karteek Addanki, Markus Saers & Dekai Wu: Improving machine translation by training against an automatic semantic frame based evaluation metric. ACL-2013: Proceedings of the 51st Meeting of the Association for Computational Linguistics, Short papers, Sofia, Bulgaria, August 4-9 2013; pp.375-381. [PDF, 370KB]

 (2013) Chi-kiu Lo & Dekai Wu: MEANT at WMT 2013: A tunable, accurate yet inexpensive semantic frame based MT evaluation metric. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.422-428. [PDF, 563KB]

(2013) Arle Richard Lommel, Aljoscha Burchardt, & Hans Uszkoreit: Multidimensional quality metrics: a flexible system for assessing translation quality.  [Aslib 2013] Translating and the Computer 35, 28-29 November 2013, etc.venues, Paddington, London, UK; 7pp. [PDF, 698KB]; presentation, 49 slides [PDF of PPT, 3999KB]

(2013) Matouš Macháček & Ondřej Bojar: Results of the WMT13 metrics shared task.  WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.45-51. [PDF, 153KB]

(2013) Lisa N.Michaud & Patricia Ann McCoy: Applying machine translation metrics to student-written translations. [NAACL HLT 2013] Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, Atlanta, Georgia, June 13 2013; pp.306-311. [PDF, 481KB]

(2013) Linda Mitchell, Johann Roturier, & Sharon O’Brien: Community-based post-editing of machine-translated content: monolingual vs. bilingual. Proceedings of MT Summit XIV Workshop on Post-editing Technology and Practice (WPTP-2), Nice, France, 2 September 2013; pp. 35-43. [PDF, 216KB]

(2013) Sudip Kumar Naskar, Antonio Toral, Federico Gaspari, & Declan Groves: Meta-evaluation of a diagnostic quality metric for machine translation. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.135-142. [PDF, 460KB]

(2013) Maja Popović, Eleftherios Avramidis, Aljoscha Burchardt, Sabine Hunsicker, Sven Schmeier, Cindy Tscherwinka, David Vilar, & Hans Uszkoreit: Learning from human judgments of machine translation output. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.231-238. [PDF, 328KB]

(2013) Akiko Sakamoto, Nayuko Watanabe, Satoshi Kamatani, & Kazuo Sumita: Development of a simultaneous interpretation system for face-to-face services and its evaluation in real situation. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.85-92. [PDF, 1073KB]

(2013) Lucia Specia: Machine translation quality evaluation and estimation [abstract]. [Aslib 2013] Translating and the Computer 35, 28-29 November 2013, etc.venues, Paddington, London, UK; presentation, 80 slides [PDF of PPT, 876KB]

(2013) Marianne Starlander & Lucía Morado Váquez: Training translation students to evaluate CAT tools using EAGLES: a case study. [Aslib 2013] Translating and the Computer 35, 28-29 November 2013, etc.venues, Paddington, London, UK; 10pp. [PDF, 603KB]; presentation, 36 slides [PDF of PPT, 615KB]

(2013) Darnes Vilariño, David Pinto, Saúl León, Yuridiana Alemán, & Helena Gómez-Adorno: BUAP: N-gram based feature evaluation for the cross-lingual textual entailment task. [SEM-2013] Second Joint Conference on Lexical and Computational Semantics, Volume 2: Seventh International Workshop on Semantic Evaluation, Atlanta, Georgia, June 14-15, 2013; pp.124-127. [PDF, 83KB]

(2013) Lingxiao Wang & Christian Boitet: Online production of HQ parallel corpora and permanent task-based evaluation of multiple MT systems: both can be obtained through iMAGs with no added cost. Proceedings of MT Summit XIV Workshop on Post-editing Technology and Practice (WPTP-2), Nice, France, 2 September 2013; pp. 103-110. [PDF, 2109KB]

(2013) Xiaofeng Wu, Hui Yu & Qun Liu: DCU participation in WMT2013 metrics task. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.435-439. [PDF, 175KB]

(2013) Rabih Zbib, Gretchen Markiewicz, Spyros Matsoukas, Richard Schwartz, & John Makhoul: Systematic comparison of professional and crowdsourced reference translations for machine translation. [NAACL-HLT 2013] The 2013 conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 9-14 June 2013, Atlanta, Georgia; pp.612-616. [PDF, 89KB]

(2012) Walid Aransa, Luong Ngoc Quang, & Antonio Toral: Diagnostic evaluation of MT with DELiC4MT. Machine Translation Marathon 2012 September 3-8, Edinburgh, UK; 14 slides [PDF of PPT, 195KB]

(2012) Eleftherios Avramidis, Aljoscha Burchardt, Christian Federmann, Maja Popović, Cindy Tscherwinka, & David  Vilar: Involving language professionals in the evaluation of machine translation. LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.1127-1130. [PDF, 320KB]]

(2012) Wilker Aziz, Sheila C.M.de Sousa, & Lucia Specia: PET: a tool for post-editing and assessing machine translation. LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.3982-3987. [PDF, 643KB]

(2012) Wilker Aziz & Lucia Specia: PET: a standalone tool for assessing machine translation through post-editing. [Aslib 2012] Translating and the Computer 34, 29-30 November 2012, One Birdcage Walk, London, UK; 5pp. [PDF, 446KB]; presentation by Lucia Specia: 54 slides [PDF, 694KB]

(2012) Wilker Aziz & Lucia Specia: PET: a tool for post-editing and assessing machine translation.  [Project paper at] EAMT 2012: Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, May 28-30 2012, ed. Mauro Cettolo, Marcello Federico, Lucia Specia, Andy Way; p.99. [PDF, 213KB]

(2012) Renu Balyan, Sudip Kumar Naskar, Antonio Toral, & Niladri Chatterjee: A diagnostic evaluation approach targeting MT systems for Indian languages. COLING 2012: Proceedings of the Workshop on Machine Translation and Parsing in Indian Languages (MTPIL-2012), Mumbai, December 2012; pp. 61-71. [PDF, 359KB]

(2012) Pratyush Banerjee, Sudip Kumar Naskar, Johann Roturier, Andy Way, & Josef van Genabith: Translation quality-based supplementary data selection by incremental update of translation models. Proceedings of COLING 2012: Technical Papers, Mumbai, December 2012; pp.149-165. [PDF, 149KB]

(2012) Nicola Bertoldi, Mauro Cettolo, Marcello Federico, & Christian Buck: Evaluating the learning curve of domain adaptive statistical machine translation systems. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.433-441. [PDF, 141KB]

(2012) Sergey Block, Michael Bloodgood, Petra Bradley, Ryan Corbett, Michael Maxwell, Erica Michael, Peter Osthus, Paul Rodrigues, & Benjamin Strauss: Evaluating parallel corpora: assessing utility for use with translation memory systems in government settings [abstract]. AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 1p. [PDF, 12KB]; presentation, 39 slides [PDF of PPT, 1589KB]

(2012) Ondřej Bojar & Dekai Wu: Towards a predicate-argument evaluation for MT. SSST-6, Sixth Workshop on Syntax, Semantics and Structure in Statistical Translation, Jeju, Republic of Korea, 12 July 2012; pp.30-38. [PDF, 126KB]

(2012) Dhouha Bouamor, Nasredine Semmar, & Pierre Zweigenbeaum: Automatic construction of a multi-word expressions bilingual lexicon: a statistical machine translation evaluation perspective. COLING 2012: Proceedings of the 3rd Workshop on Cognitive Aspects of the Lexicon (CogALex-III), Mumbai, December 2012; pp.95-107. [PDF, 210KB]

(2012) Christian Buck, Daniel Zeman, & Eva Hasler: Multiple reference translations for European languages. Machine Translation Marathon 2012 September 3-8, Edinburgh, UK; 12 slides [PDF of PPT, 83KB]

(2012) Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, & Lucia Specia: Findings of the 2012 Workshop on Statistical Machine Translation. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.10-51. [PDF, 317KB]

(2012) Julio Castillo & Paula Estrella: Semantic textual similarity for MT evaluation. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.52-58. [PDF, 278KB]

(2012) Boxing Chen, Roland Kuhn, & George Foster: Improving AMBER, an MT evaluation metric. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.59-63. [PDF, 117KB]

(2012) Boxing Chen, Roland Kuhn, & Samuel Larkin: PORT: a precision-order-recall MT evaluation metric for tuning. [ACL 2012] Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, Jeju, Republic of Korea, 8-14 July 2012; pp.930-939. [PDF, 165KB]

(2012) Jiangping Chen, Olajumoke Agozu, Wenqian Zhao, Cheng Chieh Lien, Ryan Knudson, & Ying Zhang: Integrating machine translation with digital collections for multilingual information access. AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 10pp. [PDF, 379KB]

(2012) Elisabet Comelles, Jordi Atserias, Victoria Arranz, & Irene Castellón: VERTa: linguistic features in MT evaluation.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.3944-3950. [PDF, 521KB]

(2012) Kevin Duh, Katsuhito Sudoh, Xianchao Wu, Hajime Tsukada, & Masaaki Nagata: Learning to translate with multiple objectives. [ACL 2012] Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, Jeju, Republic of Korea, 8-14 July 2012; pp.1-10. [PDF, 191KB]

(2012) Hiroshi Echizen’ya, Kenji Araki, & Eduard Hovy: Optimization for efficient determination of chunk in automatic evaluation for machine translation. COLING: Proceedings of the First International Workshop on Optimization Techniques for Human Language Technology, December 2012; pp.17-30. [PDF, 339KB]

(2012) Cristina España-Bonet, Gorka Labaka, Arantza Díaz de Ilarranza, Lluís Màrquez, & Kepa Sarasola: Deep evaluation of hybrid architectures: use of different metrics in MERT weight optimization. In: Free/Open-Source Rule-Based Machine Translation, ed.Cristina España-Bonet and Aarne  Ranta. Proceedings of a Workshop held in Gothenburg, 14-15 June, 2012; pp.65-76. [PDF, 498KB]

(2012) Christian Federmann: Appraise: an open-source toolkit for manual evaluation of MT output. Prague Bulletin of Mathematical Linguistics 98, October 2012; pp.25-35. [PDF, 487KB]

(2012) Christian Federmann: Appraise. Machine Translation Marathon 2012 September 3-8, Edinburgh, UK; 30 slides [PDF of PPT, 1216KB]

(2012) Mark Fishel, Ondřej Bojar, & Maja Popović: Terra: a collection of translation error-annotated corpora.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.7-14. [PDF, 367KB]

(2012) Mark Fishel, Rico Sennrich, Maja Popović, & Ondřej Bojar: TerrorCat: a translation error categorization-based MT quality metric. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.64-70. [PDF, 116KB]

(2012) Guillem Gascó, Martha-Alicia Rocha, Germán Sanchis-Trilles, Jesús Andrés-Ferrer, & Francisco Casacuberta: Does more data always yield better translations?  [EACL 2012] Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, Avignon, France, April 23-27, 2012; pp.152-161. [PDF,  162KB]

(2012) Meritxell Gonzàlez, Jesús Giménez & Lluís Màrquez: A graphical interface for MT evaluation and error analysis. [ACL 2012] Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, Jeju, Republic of Korea, 10 July 2012, System Demonstrations; pp.139-144. [PDF, 321KB]

(2012) Meritxell Gonzàlez & Cristina España-Bonet: New development functionality for the Asiya Suite parameter optimization with Mert. Machine Translation Marathon 2012 September 3-8, Edinburgh, UK; 9 slides [PDF of PPT, 474KB]

(2012) Yvette Graham, Timothy Baldwin, Aaron Harwood, Alistair Moffat, & Justin Zobel: Measurement of progress in machine translation. ALTA 2012: Proceedings of Australasian Language Technology Association Workshop, 4-6 December 2012, Otago University, Dunedin, New Zealand; pp. 70-78. [PDF, 576KB]

(2012) Francisco Guzman & Stephan Vogel: Understanding the performance of statistical MT systems: a linear regression framework. Proceedings of COLING 2012: Technical Papers, Mumbai, December 2012; pp.1029-1044. [PDF, 148KB]

(2012) Najeh Hajlaoui & Andrei Popescu-Belis: Translating English discourse connectives into Arabic: a corpus-based analysis and an evaluation metric. AMTA-2012: Fourth workshop on computational approaches to Arabic script-based languages. Proceedings, San Diego, November 1, 2012; pp.1-8. [PDF, 731KB]

(2012) Aaron L.F.Han, Derek F.Wong, & Lidia S.Chao: LEPOR: a robust evaluation metric for machine translation with augmented factors. Proceedings of COLING 2012: Posters, Mumbai, December 2012; pp.441-450. [PDF, 546KB]

(2012) Philipp Koehn: Simulating human judgment in machine translation evaluation campaigns. IWSLT-2012: 9th International Workshop on Spoken Language Translation, Hong Kong, December 6th-7th, 2012; pp.179-184. [PDF, 661KB]

(2012) Heinrich Kolkhorst, Kevin Kilgour, Sebastian Stüker, & Alex Waibel: Evaluation of interactive user corrections for lecture transcription. IWSLT-2012: 9th International Workshop on Spoken Language Translation, Hong Kong, December 6th-7th, 2012; pp.217-221. [PDF, 635KB]

(2012) Li LiangYou, Gong ZhengXian, & Zhou GuoDong: Phrase-based evaluation for machine translation. Proceedings of COLING 2012: Posters, Mumbai, December 2012; pp.663-672. [PDF, 201KB]

(2012) Xuansong Li, Stephanie M.Strassel, Heng Ji, Kira Griffitt, & Joe Ellis: Linguistic resources for entity linking evaluation: from monolingual to cross-lingual.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.3098-3105. [PDF, 537KB]

(2012) Ziheng Lin, Chang Liu, Hwee Tou Ng, & Min-Yen Kan: Combining coherence models and machine translation evaluation metrics for summarization evaluation. [ACL 2012] Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, Jeju, Republic of Korea, 8-14 July 2012; pp.1006-1014. [PDF, 183KB]

(2012) Chang Liu & Hwee Tou Ng: Character-level machine translation evaluation for languages with ambiguous word boundaries. [ACL 2012] Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, Jeju, Republic of Korea, 8-14 July 2012; pp.921-929. [PDF, 196KB]

(2012) Chi-kiu Lo, Anand Karthik Tumuluru, & Dekai Wu: Fully automatic semantic MT evaluation.  WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.243-252. [PDF, 379KB]

(2012) Chi-kiu Lo & Dekai Wu: Unsupervised vs. supervised weight estimation for semantic MT evaluation metrics. SSST-6, Sixth Workshop on Syntax, Semantics and Structure in Statistical Translation, Jeju, Republic of Korea, 12 July 2012; pp.49-56. [PDF, 151KB]

(2012) Adam Lopez: Putting human assessments of machine translation systems in order. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.1-9. [PDF, 112KB]

(2012) Daniel Marcu: A new method for automatic translation scoring – HyTER [abstract]. AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 1p. [PDF, 8KB]

(2012) Joseph Mariani: Language resources and evaluation for a multilingual Europe [abstract]. In: Crosslingual Language Technology in service of an integrated multilingual Europe, 4-5 May 2012, University of Hamburg, Germany; pp. 3-5. [PDF]

(2012) Marianna J.Martindale: Can statistical post-editing with a small parallel corpus save a weak MT engine?  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.2138-2142. [PDF, 353KB]

(2012) Dennis N.Mehay & Michael White: Shallow and deep paraphrasing for improved machine translation parameter optimization. AMTA-2012: Monolingual machine translation-2012 workshop. Proceedings, San Diego, November 1, 2012; pp.27-30. [PDF, 1267KB]

(2012) Alan K.Melby, Jason Housley, Paul J.Fields, & Emily Tuioti: Reliably assessing the quality of post-edited translation based on formalized structured translation specifications. AMTA-2012: Workshop on post-editing technology and practice. Proceedings, San Diego, October 28, 2012; 10pp. [PDF, 181KB]

(2012) Preslav Nakov, Francisco Guzmán, & Stephan Vogel: Optimizing for sentence-level BLEU+1 yields short translations. Proceedings of COLING 2012: Technical Papers, Mumbai, December 2012; pp.1979-1994. [PDF, 176KB]

(2012) Sharon O’Brien: Towards a dynamic quality evaluation model for translation. Journal of Specialised Translation, issue 17, January 2012; pp.55-77. [PDF, 861KB]

(2012) Maja Popović: Class error rates for evaluation of machine translation output. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.71-75. [PDF, 105KB]

(2012) Maja Popović: rgbF: an open source tool for n-gram based automatic evaluation of machine translation output. Prague Bulletin of Mathematical Linguistics 98, October 2012; pp.99-108. [PDF, 126KB]

(2012) Marion Potet, Emmanuelle Esperança-Rodier, Laurent Besacier, & Hervé Blanchon: Collection of a large database of French-English SMT output corrections.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.4043-4048. [PDF, 357KB]

(2012) Johann Roturier, Linda Mitchell, Robert Grabowski & Melanie Siegel: Using automatic machine translation metrics to analyze the impact of source reformulations. AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 10pp. [PDF, 509KB]

(2012) Michel Simard & Atsushi Fujita: A poor man’s translation memory using machine translation evaluation metrics. AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 10pp. [PDF, 958KB]

(2012) Artem Sokolov, Guillaume Wisniewski, & François Yvon: Computing lattice BLEU oracle scores for machine translation. [EACL 2012] Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, Avignon, France, April 23-27, 2012; pp.120-129. [PDF, 258KB]

(2012) Josef Steinberger & Marco Turchi: Machine translation for multilingual summary content evaluation. Proceedings of the NAACL-HLT Workshop on Evaluation Metrics and System Comparison for Automatic Summarization, Montréal, Canada, June 3-8, 2012; pp.19-27. [PDF, 126KB]

(2012) Sara Stymne & Lars Ahrenberg: On the practice of error analysis for machine translation evaluation.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.1785-1790. [PDF, 317KB]

(2012) Antonio Toral, Sudip Kumar Naskar, Federico Gaspari, & Declan Groves: DELiC4MT: a tool for diagnostic MT evaluation over user-defined linguistic phenomena. Prague Bulletin of Mathematical Linguistics 98, October 2012; pp.121-131. [PDF, 156KB]

(2012) Antonio Toral, Marc Poch, Pavel Pecina, & Gregor Thurmair: Efficiency-based evaluation of aligners for industrial applications. EAMT 2012: Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, May 28-30 2012, ed. Mauro Cettolo, Marcello Federico, Lucia Specia, Andy Way; pp.57-60. [PDF, 210KB]

(2012) Marco Turchi, Josef Steinberger, & Lucia Specia: Relevance ranking for translated texts. EAMT 2012: Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, May 28-30 2012, ed. Mauro Cettolo, Marcello Federico, Lucia Specia, Andy Way; pp.153-160. [PDF, 228KB]

(2012) Hans Uszkoreit: Cross-lingual technology: European needs and chances [abstract]. In: Crosslingual Language Technology in service of an integrated multilingual Europe, 4-5 May 2012, University of Hamburg, Germany; pp.6-7. [PDF]

(2012) Mengqiu Wang & Christopher D.Manning: Probabilistic finite state machines for regression-based MT evaluation. EMNLP-CoNLL 2012: Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Proceedings of the conference, July 12-14, Jeju Island, Korea; pp.984-994. [PDF, 613KB]

(2012) Mengqiu Wang & Christopher D.Manning: SPEDE: probabilistic edit distance metrics for MT evaluation. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.76-83. [PDF, 583KB]

(2012) Sandra Weiss & Lars Ahrenberg: Error profiling for evaluation of machine-translated text: a Polish-English case study.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.1764-1770. [PDF, 386KB]

(2012) Chris Wendt: Machine translation at Microsoft. [Aslib 2012] Translating and the Computer 34, 29-30 November 2012, One Birdcage Walk, London, UK; presentation: 39 slides [PDF, 3543KB]

(2012) Billy T.M.Wong & Chunyu Kit: Extending machine translation evaluation metrics with lexical cohesion to document level. EMNLP-CoNLL 2012: Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Proceedings of the conference, July 12-14, Jeju Island, Korea; pp.1060-1068. [PDF, 125KB]

(2012) MosesCore: Moses open source evaluation and support co-ordination for outreach and exploitation.  [Project paper at] EAMT 2012: Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, May 28-30 2012, ed. Mauro Cettolo, Marcello Federico, Lucia Specia, Andy Way; p.201. [PDF, 76KB]

(2011) Takako Aikawa & Spencer Rarrick: Are numbers good enough for you? A linguistically meaningful MT evaluation method. MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.332-337. [PDF, 145KB]

(2011) Victoria Arranz, Olivier Hamon, Karim Boudahmane, & Martine Garnier-Rizet: Protocol and lessons learnt from the production of parallel corpora for the evaluation of speech translation systems. IWSLT 2011: Proceedings of the International Workshop on Spoken Language Translation, San Francisco, December 8-9, 2011, ed. Marcello Federico, Mei-Yuh Hwang, Margit Rödder, Sebastian Stüker; pp.129-135. [PDF, 115KB]

(2011) Elefterios Avramidis, Maja Popovic, David Vilar, & Aljoscha Burchardt: Evaluate with confidence  estimation: machine ranking of translation outputs using grammatical features. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.65-70. [PDF, 107KB]

(2011) Rafael E.Banchs & Haizhou Li: AM-FM: a semantic framework for translation quality assessment. ACL-HLT 2011: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Short papers, Portland, Oregon, June 19-24, 2011; pp.153-158. [PDF, 83KB]

(2011) Luisa Bentivogli, Marcello Federico, Giovanni Moretti, & Michael Paul: Getting expert quality from the crowd for machine translation evaluation. MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.521-528. [PDF, 337KB]

(2011) Jan Berka, Martin Černý, & Ondřej Bojar: Quiz-based evaluation of machine translation. Prague Bulletin of Mathematical Linguistics, no.95, April 2011; pp.77-86. [PDF, 190KB]

(2011) Alexandra Birch & Miles Osborne: Reordering metrics for MT. ACL-HLT 2011: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, Portland, Oregon, June 19-24, 2011; pp.1027-1035. [PDF, 976KB]

(2011) Ondřej Bojar, Miloš Ercegovčević, Martin Popel, & Omar F.Zaidan: A grain of salt for the WMT manual evaluation. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.1-11. [PDF, 191KB]

(2011) Chris Callison-Burch, Philipp Koehn, Christof Monz, Omar F.Zaidan: Findings of the 2011 workshop on statistical machine translation. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.22-64. [PDF, 292KB]

(2011) Boxing Chen & Roland Kuhn: AMBER: a modified BLEU, enhanced ranking metric. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.71-77. [PDF, 148KB]

(2011) Elisabet Comelles, Irene Castellón, Jordi Atserias, Victoria Arranz, & Olivier Hamon: VERTa: exploring a multidimensional linguistically-motivated metric. [LIHMT] International Workshop on Using Linguistic Information for Hybrid Machine Translation, 18th November 2011, Universitat Politècnica de Catalunya, Barcelona; pp.21-28. [PDF, 309KB]

(2011) Daniel Dahlmeier, Chang Liu, & Hwee Tou Ng: TESLA at WMT 2011: translation evaluation and tunable metric. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.78-84. [PDF, 197KB]

(2011) Michael Denkowski & Alon Lavie: Meteor 1.3: automatic metric for reliable optimization and evaluation of machine translation systems. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.85-91. [PDF, 119KB]

(2011) Christian Federmann: How can we measure machine translation quality?  Tralogy, Paris, 3-4 March 2011;  9pp. [PDF, 252KB]

(2011) Federico Gaspari, Antonio Toral & Sudip Kumar Naskar: User-focused task-oriented MT evaluation for wikis: a case study. Proceedings of the Third Joint EM+/CNGL Workshop “Bringing MT to the User: Research Meets Translators” (JEC ’11), Luxembourg, 14 October 2011; pp.13-22. [PDF, 961KB]

(2011) Kimmo Kettunen: Packing it all up in search for a language independent MT quality measure tool [abstract]. In: Zygmunt Vetulani (ed.): Human Language Technology: Challenges for Computer Science and Linguistics: 4th Language and Technology Conference, November 6-8, Poznań, Poland (Lecture Notes in Computer Science, 6562; Heidelberg: Springer Verlag, 2011); pp.567-576. [PDF, 14KB] 

(2011) Philipp Koehn: What is a better translation? Reflections on six years of running evaluation campaigns.] Tralogy, Paris, 3-4 March 2011;  14pp. [PDF, 628KB

(2011) Gorka Labaka, Arantza Díaz de Ilarraza, Cristina España-Bonet, Kepa Sarsola, & Lluís Màrquez: Deep evaluation of hybrid architectures: simple metrics correlated with human judgments. [LIHMT] International Workshop on Using Linguistic Information for Hybrid Machine Translation, 18th November 2011, Universitat Politècnica de Catalunya, Barcelona; pp.50-57. [PDF, 313KB]

(2011) Alon Lavie: Evaluating the output of  machine translation systems. Tutorial at MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; p.15. [PDF, 75KB]

(2011) Maoxi Li, Chenqing Zong, & Hwee Tou Ng: Automatic evaluation of Chinese translation output: word-level or character-level? ACL-HLT 2011: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Short papers, Portland, Oregon, June 19-24, 2011; pp.159-164. [PDF, 151KB]

(2011) Xianhua Li, Yajuan Lü, Yao Meng, Qun Liu, & Hao Yu: Feedback selecting of manually acquired rules using automatic evaluation. [MT Summit XIII] 4th Workshop on Patent Translation, Shoichi Yokoyama (ed,), Xiamen, China, September 23, 2011; pp.52-59. [PDF, 185KB]

(2011) Shasha Liao, Cheng Wu, & Juan Huerta: Evaluating human correction quality for machine translation from crowdsourcing.  [RANLP 2011] Proceedings of Recent Advances in Natural Language Processing, Hissar, Bulgaria, 12-14 September 2011; pp.598-603. [PDF, 487KB]

(2011) Chang Liu, Daniel Dahlmeier & Hwee Tou Ng: Better evaluation metrics lead to better machine translation. [EMNLP 2011] Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Edinburgh, Scotland, UK, July 27-31, 2011; pp.375-384. [PDF, 264KB]

 (2011) Chi-kiu Lo & Dekai Wu: MEANT: an inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility via semantic frames. ACL-HLT 2011: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, Portland, Oregon, June 19-24, 2011; pp.220-229. [PDF, 935KB]

 (2011) Chi-kiu Lo & Dekai Wu: A radically simple, effective annotation and alignment methodology for semantic frame based SMT and MT evaluation. [LIHMT] International Workshop on Using Linguistic Information for Hybrid Machine Translation, 18th November 2011, Universitat Politècnica de Catalunya, Barcelona; pp.58-65. [PDF, 898KB]

(2011) Chi-kiu Lo & Dekai Wu: Structured vs. flat semantic role representations for machine translation evaluation. Proceedings of SSST-5, Fifth Workshop on Syntax, Semantics and Structure in Statistical Translation, ACL HLT 2011, Portland, Oregon, USA, June 2011; pp.10-20. [PDF, 758KB]

(2011) Yanjun Ma, Yifan He, & Josef van Genabith: From the confidence estimation of machine translation to the integration of MT and translation memory. Tutorial at MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.13-15. [PDF, 75KB]

(2011) Matouš Macháček & Ondřej Bojar: Approximating a deep-syntactic metric for MT evaluation and tuning. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.92-98. [PDF, 130KB]

(2011) Lluís Màrquez & Jesús Giménez: Automatic evaluation in machine translation: towards combined linguistically-motivated measures [abstract]. Machine Translation and Morphologically- rich Languages: Research Workshop of the Israel Science Foundation, University of Haifa, Israel, 24 January, 2011; presentation: 101 slides [PDF of PPT, 1248KB]

(2011) Erica B.Michael, Alison Blodgett, Dominic Massaro, Benjamin Bailey, Diane de Terra, Scribner Messenger, Lelyn Saner, Kathy Rhoad, Shaina Castle, & Solveig Gannon-Kurowski: The Language Product Evaluation Tool: establishing standards and developing workforce expertise. Translating and the Computer 33, 17-18 November 2011, London; 14pp. [PDF,  219KB]

(2011) John Moran & David Lewis: Unobtrusive methods for low-cost manual evaluation of machine translation. Tralogy, Paris, 3-4 March 2011;  24pp. [PDF, 244KB]

(2011) Sudip Kumar Naskar, Antonio Toral, Federico Gaspari, & Andy Way: A framework for diagnostic evaluation of MT based on linguistic checkpoints. MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.529-536. [PDF, 170KB]

(2011) Kristen Parton, Joel Tetreault, Nitin Madnani, & Martin Chodorow: E-rating machine translation. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.108-115. [PDF, 108KB]

(2011) Michael Paul & Eiichiro Sumita: Translation quality indicators for pivot-based statistical MT. [IJCNLP 2011] Proceedings of the 5th International Joint Conference on Natural Language Processing, Chiang Mai, Thailand, November 8-13, 2011; pp.811-818. [PDF, 938KB]

(2011) Daniele Pighin & Lluís Màrquez: Automatic projection of semantic structures: an application of pairwise translation ranking. Proceedings of SSST-5, Fifth Workshop on Syntax, Semantics and Structure in Statistical Translation, ACL HLT 2011, Portland, Oregon, USA, June 2011; pp.1-9. [PDF, 142KB]

 (2011) MajaPopović: Morphemes and POS tags for n-gram based evaluation metrics. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.104-107. [PDF, 68KB]

(2011) Maja Popović & Aljoscha Burchardt: From human to automatic error classification for machine translation output. [EAMT 2011]: proceedings of the 15th conference of the European Association for Machine Translation, 30-31 May 2011, Leuven, Belgium; eds. Mikel L.Forcada, Heidi Depraetere, Vincent Vandeghinste; pp.265-272. [PDF, 183KB]; presentation, 16 slides [PDF, 385KB]

(2011) Maja Popović & Hermann Ney: Towards automatic error analysis of machine translation output. Computational Linguistics 37 (4), pp. 657-688. [PDF, 219KB]

(2011) Maja Popović, David Vilar, Eleftherios Avramidis, & Aljoscha Burchardt: Evaluation without references: IBM1 scores as evaluation metrics. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.99-103. [PDF, 88KB]

 (2011) Miguel Rios, Wilker Aziz, & Lucia Specia: TINE: a metric to assess MT adequacy. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.116-122. [PDF, 132KB]

(2011) Felipe Sánchez-Martínez: Choosing the best machine translation system to translate a sentence by using only source-language information. [EAMT 2011]: proceedings of the 15th conference of the European Association for Machine Translation, 30-31 May 2011, Leuven, Belgium; eds. Mikel L.Forcada, Heidi Depraetere, Vincent Vandeghinste; pp.97-104. [PDF, 338KB] ; presentation, 32 slides [PDF]

(2011) Raivis Skadiņš, Maris Puriņš, Inguna Skadiņa, & Andrejs Vasiļjevs: Evaluation of SMT in localization to under-resourced inflected language. [EAMT 2011]: proceedings of the 15th conference of the European Association for Machine Translation, 30-31 May 2011, Leuven, Belgium; eds. Mikel L.Forcada, Heidi Depraetere, Vincent Vandeghinste; pp.35-40. [PDF, 287KB]; presentation, 17 slides [PDF, 796KB]

 (2011) Xingyi Song & Trevor Cohn: Regression and ranking based optimisation for sentence level machine translation evaluation. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.123-129. [PDF, 122KB]

(2011) Lucia Specia: Linguistic indicators for quality estimation of machine translation [abstract]. [LIHMT] International Workshop on Using Linguistic Information for Hybrid Machine Translation, 18th November 2011, Universitat Politècnica de Catalunya, Barcelona; p.14. [PDF, 139KB]

(2011) Lucia Specia, Najeh Hajlaoui, Catalina Hallett, & Wilker Aziz: Predicting machine translation adequacy. MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.513-520. [PDF, 193KB]

 (2011) Lucia Specia: Quality estimation for machine translation. Proceedings of the Third Joint EM+/CNGL Workshop “Bringing MT to the User: Research Meets Translators” (JEC ’11), Luxembourg, 14 October 2011; 46 slides [PDF of PPT, 749KB]

(2011) Ankit K.Srivastava, Yanjun Ma, & Andy Way: Oracle-based training for phrase-based statistical machine translation. [EAMT 2011]: proceedings of the 15th conference of the European Association for Machine Translation, 30-31 May 2011, Leuven, Belgium; eds. Mikel L.Forcada, Heidi Depraetere, Vincent Vandeghinste; pp.169-176. [PDF, 428KB]

 (2011) Marianne Starlander & Paula Estrella: Looking for the best evaluation method for interlingua-based spoken language translation in the medical domain. Proceedings of the 8th international NLPSC workshop. Special theme: Human-machine interaction in translation, Copenhagen Business School, 20-21 August 2011; ed.Bernadette Sharp, Michael Zock, Michael Carl, Arnt Lykke Jakobsen (Copenhagen Studies in Language 41), Frederiksberg: Samfundslitteratur, 2011; pp.81-92. [PDF, 905KB]

 (2011) Hirokazu Suzuki: Automatic post-editing based on SMT and its selective application by sentence-level automatic quality evaluation. MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.156-163. [PDF, 725KB]

 (2011) David Talbot, Hideto Kazawa, Hiroshi Ichikawa, Jason Katz-Brown, Masakazu Seno, & Franz J.Och: A lightweight evaluation framework for machine translation reordering. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.12-21. [PDF, 371KB]

 (2011) Billy Wong & Chunyu Kit: Comparative evaluation of term informativeness measures in machine translation evaluation metrics. MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.537-544. [PDF, 147KB]

 (2011) Tong Xiao, Jingbo Zhu, Shujie Yao, & Hao Zhang: Document-level consistency verification in machine translation. MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.131-138. [PDF, 126KB]

 (2011) Omar F.Zaidan: MAISE: a flexible, configurable, extensible open source package for mass AI system evaluation. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.130-134. [PDF, 108KB]

 (2011) Dan Zeman, Mark Fishel, Jan Berka, & Ondřej Bojar: Addicter: what’s wrong with my translations? Sixth Machine Translation Marathon, 5-10 September 2011, Trento; Prague Bulletin of Mathematical Linguistics, no.96, October 2011; pp.79-88. [PDF, 276KB]; presentation, 20 slides [PDF of PPT, 359KB]

(2011) ACCURAT: analysis and evaluation of comparable corpora for under resourced areas of machine translation. (European Machine Translation Projects.) [EAMT 2011]: proceedings of the 15th conference of the European Association for Machine Translation, 30-31 May 2011, Leuven, Belgium; eds. Mikel L.Forcada, Heidi Depraetere, Vincent Vandeghinste; p.323. [PDF, 287KB]

(2010) Milam Aiken & Mina Park: The efficacy of round-trip translation for MT evaluation. Translation Journal 14 (1), January 2010; 10pp. [PDF, 149KB]

(2010) Marianna Apidianaki & Yifan He: An algorithm for cross-lingual sense-clustering tested in a MT evaluation setting. Proceedings of the 7th International Workshop on Spoken Language Translation, 2-3 December 2010, Paris, France; pp.219-226. [PDF, 474KB]

(2010) Alexandra Birch & Miles Osborne: LRscore for evaluating lexical and reordering quality in MT.  ACL 2010: Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR. Proceedings of the workshop, 15-16 July 2010, Uppsala University, Uppsala, Sweden; pp. 327-332. [PDF, 123KB]

(2010) Michael Bloodgood & Chris Callison-Burch: Using Mechanical Turk to build machine translation evaluation sets. Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, Los Angeles, CA, June 2010; pp.208-211. [PDF, 89KB]

(2010) Ondřej Bojar, Kamil Kos, & David Mareček: Tackling sparse data issue in machine translation evaluation. ACL 2010: the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, July 11-16, 2010: Proceedings of the Conference Short Papers; pp.86-91. [PDF, 132KB]

(2010) Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, & Omar F.Zaidan: Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation. ACL 2010: Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR. Proceedings of the workshop, 15-16 July 2010, Uppsala University, Uppsala, Sweden; pp. 17-53. [PDF, 837KB]

(2010) Marine Carpuat & Mona Diab: Task-based evaluation of multiword expressions: a pilot study in statistical machine translation. NAACL HLT 2010: Human Language Technologies: the 2010 annual conference of the North American Chapter of the Association for Computational Linguistics. Proceedings… June 2-4, 2010, Los Angeles, California; pp.242-245. [PDF, 92KB]

(2010) Daniel Cer, Christopher D.Manning, & Daniel Jurafsky: The best lexical metric for phrase-based statistical MT system optimization. NAACL HLT 2010: Human Language Technologies: the 2010 annual conference of the North American Chapter of the Association for Computational Linguistics. Proceedings… June 2-4, 2010, Los Angeles, California; pp.555-563. [PDF, 169KB]

(2010) Michael Denkowski & Alon Lavie: Choosing the right evaluation for machine translation: an examination of annotator and automatic metric performance on human judgment tasks. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; 9pp. [PDF, 151KB]

(2010) Michael Denkowski & Alon Lavie: Extending the METEOR machine translation evaluation metric to the phrase level. NAACL HLT 2010: Human Language Technologies: the 2010 annual conference of the North American Chapter of the Association for Computational Linguistics. Proceedings… June 2-4, 2010, Los Angeles, California; pp.250-253. [PDF, 99KB]

(2010) Michael Denkowski & Alon Lavie: Exploring normalization techniques for human judgments of machine translation adequacy collected using Amazon Mechanical Turk. Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, Los Angeles, CA, June 2010; pp.57-61. [PDF, 248KB]

(2010) Michael Denkowski & Alon Lavie: METEOR-NEXT and the METEOR paraphrase tables: improved evaluation support for five target languages. ACL 2010: Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR. Proceedings of the workshop, 15-16 July 2010, Uppsala University, Uppsala, Sweden; pp. 339-342. [PDF, 104KB]

(2010) Marcus Dobrinkat, Tero Tapiovaara, Jaakko Väyrynen, & Kimmo Kettunen: Evaluating machine translations using mNCD. ACL 2010: the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, July 11-16, 2010: Proceedings of the Conference Short Papers; pp.80-85. [PDF, 145KB]

(2010) Marcus Dobrinkat, Jaako Väyrynen, Tero Tapiovaara, & Kimmo Kettunen: Normalized compression distance based measures for MetricsMATR 2010. ACL 2010: Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR. Proceedings of the workshop, 15-16 July 2010, Uppsala University, Uppsala, Sweden; pp. 343-348. [PDF, 118KB]

(2010) Jinhua Du & Andy Way: Using TERp to augment the system combination for SMT. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; 9pp. [PDF, 177KB]

(2010) Hiroshi Echizen-ya & Kenji Araki: Automatic evaluation method for machine translation using noun-phrase chunking. ACL 2010: the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, July 11-16, 2010: Conference proceedings; pp.108-117. [PDF, 209KB]

(2010) Mireia Farrús, Marta R.Costa-jussà, José B.Mariño, & José A.R.Fonollosa: Linguistic-based evaluation criteria to identift statistical machine translation errors. EAMT 2010: Proceedings of the 14th Annual conference of the European Association for Machine Translation, 27-28 May 2010, Saint-Raphaël, France. Proceedings ed.Viggo Hansen and François Yvon; 7pp. [PDF, 606KB]

(2010) Christian Federmann: Appraise: an open-source toolkit for manual phrase-based evaluation of translations. LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.1731-1734. [PDF, 368KB]

(2010) Pamela Forner, Danilo Giampiccolo, Bernardo Magnini, Anselmo Peñas, Álvaro Rodrigo, & Richard Sutcliffe: Evaluating multilingual question answering systems at CLEF. LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.2774-2781. [PDF, 314KB]

(2010) Jesús Giménez & Lluís Màrquez: Asiya: an open toolkit for automatic machine translation (meta-)evaluation. Fifth Machine Translation Marathon, 13-18 September, University of Le Mans; Prague Bulletin of Mathematical Linguistics, no.94, September 2010; pp.77-86. [PDF, 271KB]; presentation [PDF, 299KB]

(2010) Jesús Giménez: Empirical machine translation and its evaluation. Presentation of Best Thesis Award for 2008 at EAMT 2010: 14th Annual conference of the European Association for Machine Translation, 28 May 2010, Saint-Raphaël, France. 39 slides [PDF of PPT presentation, 496KB]

(2010) Jesús Giménez, Lluís Màrquez, Elisabet Comelles, Irene Castellón, & Victoria Arranz: Document-level automatic MT evaluation based on discourse representations. ACL 2010: Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR. Proceedings of the workshop, 15-16 July 2010, Uppsala University, Uppsala, Sweden; pp. 333-338. [PDF, 112KB]

(2010) Yvette Graham & Josef van Genabith: Factor templates for factored machine translation models. Proceedings of the 7th International Workshop on Spoken Language Translation, 2-3 December 2010, Paris, France; pp.275-282. [PDF, 366KB]

(2010) Olivier Hamon: Is my judge a good one? LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.1735-1740. [PDF, 281KB]

(2010) Yifan He, Jinhua Du, Andy Way, & Josef van Genabith: The DCU dependency-based metric in WMT-MetricsMATR 2010. ACL 2010: Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR. Proceedings of the workshop, 15-16 July 2010, Uppsala University, Uppsala, Sweden; pp. 349-353. [PDF, 444KB]

 (2010) Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, & Hajime Tsukada: Automatic evaluation of translation quality for distant language pairs.  [EMNLP 2010] Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, MIT, Massachusetts, USA, 9-11 October 2010; pp.778-788. [PDF, 226KB]

(2010) Judith L. Klavans: [Panel on] Task-based evaluation methods for machine translation, in practice and theory. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; abstract

(2010) Adrien Lardilleux, Julien Gosme, & Yves Lepage: Bilingual lexicon induction: effortless evaluation of word alignment tools and production of resources for improbable language pairs. [LREC 2010]: Proceedings of the Second Workshop on African Language Technology, AFLAT 2010, Valetta, Malta; pp.252-256. [PDF, 330KB]

(2010) Alon Lavie: Evaluating the output of machine translation systems. Tutorial at AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31, 2010; 86 slides [PDF of PPT, 956KB]

(2010) Chang Liu, Daniel Dahlmeier, & Hwee Tou Ng: TESLA: translation evaluation of sentences with linear-programming-based analysis. ACL 2010: Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR. Proceedings of the workshop, 15-16 July 2010, Uppsala University, Uppsala, Sweden; pp. 354-359. [PDF, 181KB]

(2010) Chi-kiu Lo & Dekai Wu: Evaluating machine translation utility via semantic role labels. LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.2873-2877. [PDF, 417KB]

(2010) Chi-kiu Lo & Dekai Wu: Semantic vs. syntactic vs. n-gram structure for machine translation evaluation. SSST-4: Proceedings of Fourth Workshop on Syntax and Structure in Statistical Translation, ed. Dekai Wu. COLING 2010/SIGMT Workshop, 23rd International Conference on Computational Linguistics, Beijing, China, 28  August 2010; pp.52-60. [PDF, 401KB]

(2010) Adam Lopez: Introduction and machine translation evaluation. Presentation at Fifth Machine Translation  Marathon, 13-18 September 2010, University of Le Mans; 329 slides. [PDF, 11,952KB]

(2010) Daniel Marcu, Kathleen Egan, Chuck Simmons, & Ning-Ning Mahlmann: Utilizing automated translation with quality scores to increase productivity. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; abstract

(2010) Aurélien Max, Josep Maria Crego, & François Yvon: Contrastive lexical evaluation of machine translation. LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.1753-1757. [PDF, 400KB]

(2010) Sergio Penkale, Yanjun Ma, Daniel Galron, & Andy Way: Accuracy-based scoring for phrase-based statistical machine translation. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; 10pp. [PDF, 281KB]

(2010) Horacio Saggion, Juan-Manuel Torres-Moreno, Iria da Cunha, Eric SanJuan, & Patricia Velázquez-Morales: Multilingual summarization evaluation without human models. Coling 2010: 23rd International Conference on Computational Linguistics, 23-27 August 2010, Beijing International Convention Center, Beijing, China, Posters volume; pp.1059-1067. [PDF, 238KB]

(2010) Anne H.Schneider, Ielka van der Sluis, & Saturnino Luz: Comparing intrinsic and extrinsic evaluation of MT output in a dialogue system. Proceedings of the 7th International Workshop on Spoken Language Translation, 2-3 December 2010, Paris, France; pp.329-336. [PDF, 448KB]

(2010) Inguna Skadiņa, Andrejs Vasiļjevs, Raivis Skadiņš, Robert Gaizauskas, Dan Tufiş, & Tatiana Gornostay: Analysis and evaluation of comparable corpora for under resourced areas of machine translation. [LREC 2010] Proceedings of the 3rd Workshop on Building and Using Comparable Corpora, Malta, 22 May 2010; pp.6-14. [PDF, 318KB]

(2010) Lucia Specia & Jesús Giménez: Combining confidence estimation and reference-based metrics for segment-level MT evaluation. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; 10pp. [PDF, 152KB]

(2010) Lucia Specia, Nicola Cancedda, & Marc Dymetman: A dataset for assessing machine translation evaluation metrics. LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.3375-3378. [PDF, 362KB]

(2010) Kathrin Spreyer: Notes on the evaluation of dependency parsers obtained through cross-lingual projection. Coling 2010: 23rd International Conference on Computational Linguistics, 23-27 August 2010, Beijing International Convention Center, Beijing, China, Posters volume; pp.1176-1184. [PDF, 114KB]

(2010) Osamuyimen Stewart, David Lubensky, Scott Macdonald, & Julie Marcotte: Using machine translation for localization of electronic support content: evaluating end-user satisfaction. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; 6pp. [PDF, 39KB]

(2010) Sara Stymne & Lars Ahrenberg: Using a grammar checker for evaluation and postprocessing of statistical machine translation. LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.2175-2181. [PDF,

(2010) Yanli Sun: Mining the correlation between human and automatic evaluation at sentence level. LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.1726-1730. [PDF, 619KB]

(2010) Irina Temnikova: Cognitive evaluation approach for controlled language post-editing experiment. LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.3485-3490. [PDF, 631KB]

(2010) Carol Van Ess-Dykema, Jocelyn Phillips, Florence Reeder, & Laurie Gerber: Paralinguist assessment decision factors for machine translation output: a case study. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; 7pp. [PDF, 180KB]; abstract

 (2010) Guillaume Wisniewski, Alexandre Allauzen, & François Yvon: Assessing phrase-based translation models with oracle decoding. [EMNLP 2010] Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, MIT, Massachusetts, USA, 9-11 October 2010; pp.933-943. [PDF, 188KB]

(2010) Billy T-M Wong & Chunyu Kit: The parameter-optimized ATEC metric for MT evaluation. ACL 2010: Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR. Proceedings of the workshop, 15-16 July 2010, Uppsala University, Uppsala, Sweden; pp. 360-364. [PDF, 114KB]

(2010) Billy Tak-Ming Wong: Semantic evaluation of machine translation.  LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.2884-2888. [PDF, 574KB]

(2010) Deyi Xiong, Min Zhang, & Haizhou Li: Error detection for statistical machine translation using linguistic features. ACL 2010: the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, July 11-16, 2010: Conference proceedings; pp.604-611. [PDF, 520KB]

(2010) Omar F.Zaidan & Chris Callison-Burch: Predicting human-targeted translation edit rate via untrained human annotators. NAACL HLT 2010: Human Language Technologies: the 2010 annual conference of the North American Chapter of the Association for Computational Linguistics. Proceedings… June 2-4, 2010, Los Angeles, California; pp.369-372. [PDF, 206KB]

(2010) Junguo Zhu, Muyun Yang, Bo Wang, Sheng Li, & Tiejun Zhao: All in strings: a powerful string-based automatic MT evaluation metric with multiple granularities. Coling 2010: 23rd International Conference on Computational Linguistics, 23-27 August 2010, Beijing International Convention Center, Beijing, China, Posters volume; pp.1533-1540. [PDF, 159KB]

Evaluations of systems (see also User evaluation)

(2015) Miguel Á.Candel-Mora: Evaluation of English to Spanish MT output of Tourism 2.0 consumer-generated reviews with post-editing purposes. Proceedings of the 37th Conference Translating and the Computer, London, November 26-27, 2015; pp.37-47. [PDF, 284KB]

(2015) Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, Roldano Cattoni, & Marcello Federico: The IWSLT 2015 Evaluation Campaign. [IWSLT 2015] Proceedings of the International Workshop on Spoken Language Translation, December 3-4, 2015, Da Nang, Vietnam; pp. 2-14. [PDF, 3.9MB]

(2015) Alan K.Melby: Quality evaluation of four translations of a kidney document: focus on reliability. MT Summit XV, October 30 – November 3, 2015, Miami, Florida, USA. Proceedings of MT Summit XV: vol.2: MT Users’ Track; pp.65-68. [PDF, 547KB]

(2015) Mark Seligman & Mike Dillinger: Evaluation and revision of a speech translation system for healthcare. [IWSLT 2015] Proceedings of the International Workshop on Spoken Language Translation, December 3-4, 2015, Da Nang, Vietnam; pp.209-216. [PDF, 3.1MB]

(2014) Nora Aranberri, Gorka Labaka, Arantza Diaz de Ilarraza, & Kepa Sarasola Comparison of post-editing productivity between professional translators and lay users. AMTA 2014: proceedings of the eleventh conference of the  Association for Machine Translation in the Americas, Vancouver, BC, October 22-26; Workshop on Post-editing Technology and Practice (WPTP-3); pp.20-33. [PDF, 247KB]

(2014) A.Barreiro, J.Monti, B.Orliac, S. Preuß, K.Arrieta, W.Ling, F.Batista & I.Trancoso: Linguistic evaluation of support verb constructions by OpenLogos and Google Translate. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.35-40. [PDF, 131KB]

(2014) Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, & Aleš Tamchyna: Findings of the 2014 Workshop on Statistical Machine Translation. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.12-58. [PDF, 642KB]

(2014) Laura Casanellas & Lena Marg: Assumptions, expectations and outliers in post-editing. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; pp.93-97. [PDF, 592KB]

(2014) Sheila Castilho, Sharon O’Brien, Fabio Alves, & Morgan O’Brien: Does post-editing increase usability? A study with Brazilian Portuguese as target language. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; pp.183-190. [PDF, 754KB]

(2014) Thierry Etchegoyhen, Lindsay Bywood, Mark Fishel, Panayota Georgakopoulou, Jie Jiang, Gerard van Loenhout, Arantza del Pozo, Mirjam Sepesy Maučec, Anja Turner, & Martin Volk: Machine translation for subtitling: a large-scale evaluation. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.46-53. [PDF, 301KB]

(2014) Kevin Flanagan: Filling in the gaps: what we need from TM subsegment recall. Translating and the Computer 36: proceedings. Asling: International Society for Advancement in Language Technology, 27-28 November 2014; pp.12-23. [PDF, 508KB]

(2014) Fabrizio Gotti, Philippe Langlais, & Atefeh Farzindar: Hashtag occurrences, layout and translation: a corpus-driven analysis of tweets published by the Canadian government.  LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.2254-2261. [PDF, 486KB]

(2014) Rohit Gupta, Hanna Bechara, & Constantin Orasan: Intelligent translation memory matching and retrieval metric exploiting linguistic technology. Translating and the Computer 36: proceedings. Asling: International Society for Advancement in Language Technology, 27-28 November 2014; pp.86-89. [PDF, 176KB]

(2014) Jeffrey Killman: Vocabulary accuracy of statistical machine translation in the legal context. AMTA 2014: proceedings of the eleventh conference of the Association for Machine Translation in the Americas, Vancouver, BC, October 22-26; Workshop on Post-editing Technology and Practice (WPTP-3); pp.85-98. [PDF, 205KB]

(2014) Jianri Li, Se-Jong Kim, Hwidong Na, & Jong-Hyeok Lee: Postech’s system description for medical text translation task. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.229-232. [PDF, 262KB]

(2014) Arle Lommel, Aljoscha Burchardt, Maja Popović, Kim Harris, Eleftherios Avramidis, & Hans Uszkoreit: Using a new analytic measure for the annotation and analysis of MT errors on real data. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; pp.165-172. [PDF, 1781KB]

(2014) Mārcis Pinnis, Raivis Skadiņš, & Andrejs Vasiļjevs: Real-world challenges in application of MT for localization: the Baltic case. AMTA 2014: proceedings of the eleventh conference of the Association for Machine Translation in the Americas, Vancouver, BC, October 22-26; Commercial MT Users, pp.66-79. [PDF, 1540KB]

(2014) Jayendra Rakesh Yeka, Prasanth Kolachina, & Dipti Misra Sharma: Benchmarking of English-Hindi parallel corpora. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.1812-1818. [PDF, 627KB]

(2014) Keisuke Sakaguchi, Matt Post, & Benjamin Van Durme: Efficient elicitation of annotations for human evaluation of machine translation. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.1-11. [PDF, 641KB]

(2014) Violeta Seretan, Pierrette Bouillon, & Johanna Gerlach: A large-scale evaluation of pre-editing strategies for improving user-generated content translation. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.1793-1799. [PDF, 476KB]

(2014) Antonio Toral, Raphael Rubino, Miquel Esplà-Gomis, Tommi Pirinen, Andy Way, & Gema  Ramírez-Sánchez: Extrinsic evaluation of web-crawlers in machine translation: a study on Croatian-English for the tourism domain. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; pp. 221-224. [PDF, 345KB]

(2014) Zdeňka Urešová, Ondřej Dušek, Jan Hajič, & Pavel Pecina: Multilingual test sets for machine translation of search queries for cross-lingual information retrieval in the medical domain. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.3244-3247. [PDF, 116KB]

 (2014) SCATE: Smart Computer Aided Translation Environment. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; p.135. [PDF, 311KB]

(2013) Proceedings of the 10th International Workshop on Spoken Language Translation, Heidelberg, Germany, Dec.5-6, 2013.

(2013) Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut & Lucia Specia: Findings of the 2013 Workshop on Statistical Machine Translation. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.1-44. [PDF, 1767KB]

(2013) Pavel Braslavski, Alexander Beloborodov, Maxim Khalilov, & Serge Sharoff: English→Russian MT evaluation campaign. ACL-2013: Proceedings of the 51st Meeting of the Association for Computational Linguistics, Short papers, Sofia, Bulgaria, August 4-9 2013; pp.262-267. [PDF, 162KB]

(2013) Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, & Marcello Federico: Report on the 10th IWSLT evaluation campaign. [IWSLT 2013] Proceedings of the 10th International Workshop on Spoken Language Translation, Heidelberg, Germany, Dec.5-6, 2013; 18pp. [PDF, 270KB]

(2013) Marta R.Costa-jussà, Rafael E.Banchs, Reinhard Rapp, Patrik Lambert, Kurt Eberle, & Bogdan Babych: Workshop on Hybrid Approaches to Translation: overview and developments. Proceedings of the Second Workshop on Hybrid Approaches to Translation, Sofia, Bulgaria, August 8, 2013; pp. 1-6. [PDF, 235KB]

(2013) Lluís Formiga, Lluís Màrquez, & Jaume Pujantell: Real-life translation quality estimation for MT system selection. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.69-76. [PDF, 536KB]

(2013) Lluís Formiga, Meritxell Gonzàlez, Alberto Barrón-Cedeño, José A. R. Fonollosa & Lluís Màr­quez: The TALP-UPC approach to system selection: Asiya features and pairwise classification using ran­dom forests. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.359-364. [PDF, 181KB]

(2013) Mark Hopkins & Jonathan May: Models of translation competitions. ACL-2013: Proceedings of the 51st Meeting of the Association for Computational Linguistics, Sofia, Bulgaria, August 4-9 2013; pp.1416-1425. [PDF, 862KB]

(2013) Lucas Nunes Vieira: An evaluation of tools for post-editing research: the current picture and future needs. Proceedings of MT Summit XIV Workshop on Post-editing Technology and Practice (WPTP-2), Nice, France, 2 September 2013; pp. 93-101. [PDF, 612KB]

(2013) Carlos Ramish, Laurent Besacier, & Alexander Kobzar: How hard is it to automatically translate phrasal verbs from English to French?  [MT Summit XIV] Proceedings of the Workshop on Multi-word Units in Machine Translation and Translation Technology, Nice, September 3, 2013; pp.53-61. [PDF, 303KB]

(2013) Akiko Sakamoto, Kazuhiko Abe, Kazuo Sumita & Satoshi Kamatani: Evaluation of a simultaneous interpretation system and analysis of speech log for user experience assessment. [IWSLT 2013] Proceedings of the 10th International Workshop on Spoken Language Translation, Heidelberg, Germany, Dec.5-6, 2013; 8pp. [PDF, 691KB]

(2013) Nathalie de Sutter & Joachim van den Bogaert: Quality or productivity? Let’s do both!  Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.381-390. [PDF of PPT, 824KB]

(2013) Lori Thicke: A technology agnostic approach to machine translation. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.309-311. [PDF, 356KB]

(2013) Antonio Toral: Hybrid selection of language model training data using linguistic information and perplexity. Proceedings of the Second Workshop on Hybrid Approaches to Translation, Sofia, Bulgaria, August 8, 2013; pp.8-12. [PDF, 307KB]

(2013) Omar Zaidan & Vishal Chowdhary: Evaluating (and improving) sentence alignment under noisy conditions. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.484-493. [PDF, 194KB]

(2012) Marianna Apidianaki: Measuring the adequacy of cross-lingual paraphrases in a machine translation setting. Proceedings of COLING 2012: Posters, Mumbai, December 2012; pp.63-72. [PDF, 137KB]

(2012) Eleftherios Avramidis: Comparative quality estimation: automatic sentence-level ranking of multiple machine translation outputs. Proceedings of COLING 2012: Technical Papers, Mumbai, December 2012; pp.115-132. [PDF, 235KB]

(2012) Hanna Béchara, Raphaël Rubino, Yifan He, Yanjun Ma, & Josef van Genabith: An evaluation of statistical post-editing systems applied to RBMT and SMT systems. Proceedings of COLING 2012: Technical Papers, Mumbai, December 2012; pp.215-230. [PDF, 243KB]

(2012) Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, & Lucia Specia: Findings of the 2012 Workshop on Statistical Machine Translation. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.10-51. [PDF, 317KB]

(2012) Sachiyo Demizu & Mike Holland: Linguists love art and management loves efficiency – can MT be the solution? AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 10pp. [PDF, 106KB]

(2012) Atefeh Farzindar & Wael Khreich: Evaluation of domain adaptation techniques for TRANSLI in a real-world environment. AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 5pp. [PDF, 136KB]

(2012) M.Federico, M.Cettolo, L.Bentivogli, M.Paul, & S.Stüker: Overview of the IWSLT 2012 evaluation campaign. IWSLT-2012: 9th International Workshop on Spoken Language Translation, Hong Kong, December 6th-7th, 2012; pp.12-33. [PDF, 772KB]; presentation: 44 slides [PDF of PPT, 3701KB]

(2012) Marcello Federico, Sebastian Stüker, Luisa Bentivogli, Michael Paul, Mauro Cettolo, Teresa Herrmann, Jan Niehues, & Giovanni Moretti: The IWSLT 2011 evaluation campaign on automatic talk translation.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.3543-3540. [PDF, 411KB]

(2012) Christian Federmann, Eleftherios Avramidis, Marta R.Costa-jussà, Maite Melero, Pavel Pecina, & Josef van Genabith: The ML4HMT workshop on optimising the division of labour in hybrid machine translation.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.3430-3435. [PDF, 631KB]

(2012) Weimin Jiang: Engine-specific Chinese-English user parallel corpora AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 9pp. [PDF, 326KB]; presentation, 25 slides [PDF of PPT,540KB]

(2012) Rasoul Samad Zadeh Kaljahi, Raphael Rubino, Johann Roturier, & Jennifer Foster: A detailed analysis of phrase-based and syntax-based machine translation: the search for systematic differences. AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 10pp. [PDF, 140KB]

(2012) Philipp Koehn: Computer-aided translation. Machine Translation Marathon 2012 September 3-8, Edinburgh, UK; 118 slides [PDF of PPT, 958KB]

(2012) Philipp Koehn: Simulating human judgment in machine translation evaluation campaigns. IWSLT-2012: 9th International Workshop on Spoken Language Translation, Hong Kong, December 6th-7th, 2012; pp.179-184. [PDF, 661KB]

(2012) Keith J.Miller, Elizabeth Schroeder Richerson, Sarah McLeod, James Finley, & Aaron Schein: International multicultural name matching competition: design, execution, results, and lessons learned.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.3111-3117. [PDF, 638KB]

(2012) Tsuyoshi Okita: Annotated corpora for word alignment between Japanese and English and its evaluation with MAP-based word aligner.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.3241-3248. [PDF, 522KB]

(2012) Kristen Parton, Nizar Habash, & Kathleen McKeown: Lost and found in translation: the impact of machine translated results on translingual information retrieval. AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 10pp. [PDF, 238KB]

(2012) Daniele Pighin, Lluís Formiga, & Lluís Màrquez: A graph-based strategy to streamline translation quality assessments. AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 10pp. [PDF, 243KB]

(2012) Alexandros Poulis & David Kolovratnik: To post-edit or not to post-edit? Estimating the benefits of MT post-editing for a European organization. AMTA-2012: Workshop on post-editing technology and practice. Proceedings, San Diego, October 28, 2012; 9pp. [PDF, 390KB]

(2012) Laura Ramírez Polo: Use and evaluation of controlled languages in industrial environments and feasibility study for the implementation of machine translation. PhD thesis, Universidad de Valencia, June 2012; 504pp. [PDF, 6696KB]

(2012) Lorenza Russo, Sharid Loáiciga, & Asheesh Gulati: Italian and Spanish null subjects: a case study evaluation in an MT perspective.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.1779-1784. [PDF, 372KB]

(2012) Sanja Seljan, Tomislav Vičić, & Marija Brkić: BLEU evaluation of machine-translated English-Croatian legislation.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.2143-2148. [PDF, 332KB]

(2012) Chunqi Shi, Donghui Lin, Masahiko Shimada, & Toru Ishida: Two phase evaluation for selecting machine translation services.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.1771-1778. [PDF, 645KB]

(2012) Radu Soricut & Sushant Narsale: Combining quality prediction and system selection for improved automatic translation output. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.163-170. [PDF, 135KB]

(2012) Sara Stymne, Henrik Danielsson, Sofia Bremin, Hongzhan Hu, Johanna Karlsson, Anna Prytz Lillkull, & Martin Wester: Eye tracking as a tool for machine translation error analysis.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.1121-1126. [PDF, 357KB]

(2012) George Tambouratzis, Sokratis Sofianopoulos, & Marina Vassiliou: Evaluating the translation accuracy of a novel language-independent MT methodology. Proceedings of COLING 2012: Technical Papers, Mumbai, December 2012; pp.2569-2583. [PDF, 340KB]

(2012) John Tinsley, Alexandru Ceausu, Jian Zhang, Heidi Depraetere, & Joeri Van de Walle: IPTranslator: facilitating patent search with machine translation. AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 9pp. [PDF, 419KB]

(2012) Trond Trosterud & Kevin Brubeck Unhammer: Evaluating North Sámi to Norwegian assimilation RBMT. In: Free/Open-Source Rule-Based Machine Translation, ed.Cristina España-Bonet and Aarne  Ranta. Proceedings of a Workshop held in Gothenburg, 14-15 June, 2012; pp.13-26. [PDF, 570KB]

(2012) Špela Vintar, Darja Fišer, & Aljoša Vrščaj: Were the clocks striking or surprising? Using WSD to improve MT performance. EACL Joint Workshop on Exploiting Synergies between Information Retrieval and Machine Translation (ESIRMT) and Hybrid Approaches to Machine Translation (HyTra): Proceedings of the workshop, 23-24 April 2012, Avignon, France; pp.87-92. [PDF, 283KB]

(2012) Min Zhang, Haizhou Li, Ming Liu, & A.Kumaran: Whitepaper of NEWS 2012 shared task on machine transliteration. [ACL 2012] Proceedings of NEWS 2012 Named Entities Workshop, July 12, 2012, Jeju, Republic of Korea; pp.1-9. [PDF, 192KB]

(2012) Min Zhang, Haizhou Li, A.Kumaran, & Ming Liu: Report of NEWS 2012 machine transliteration shared task. [ACL 2012] Proceedings of NEWS 2012 Named Entities Workshop, July 12, 2012, Jeju, Republic of Korea; pp.10-20. [PDF, 136KB]

(2011) Takako Aikawa & Spencer Rarrick: Are numbers good enough for you? A linguistically meaningful MT evaluation method. MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.332-337. [PDF, 145KB]

(2011) Rafael E.Banchs: Comparative evaluation of Spanish segmentation strategies for Spanish-Chinese transliteration. [IJCNLP 2011] Proceedings of the 2011 Named Entities Workshop, Chiang Mai, Thailand, November 12, 2011; pp.41-48. [PDF, 198KB]

(2011) Frédéric Blain, Jean Senellart, Holger Schwenk, Mirko Plitt, & Johann Roturier: Qualitative analysis of post-editing for high quality machine translation. MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.164-171. [PDF, 218KB]

(2011) Karim Boudahmane, Bianka Buschbeck, Eunach Cho, Josep Maria Crego, Markus Freitag, Thomas Lavergne, Hermann Ney, Jan Niehues, Stephan Peitz, Jean Senellart, Artem Sokolov, Alex Waibel, Tonio Wandmacher, Joern Wuebker, & François Yvon: Advances on spoken language translation in the Quaero program. IWSLT 2011: International Workshop on Spoken Language Translation, San Francisco, December 8-9, 2011; pp.114-120. [PDF, 195KB]

(2011) Marija Brkić, Sanja Seljan, & Maja Matetić: Machine translation evaluation for Croatian-English and English-Croation language pairs. Proceedings of the 8th international NLPSC workshop. Special theme: Human-machine interaction in translation, Copenhagen Business School, 20-21 August 2011; ed.Bernadette Sharp, Michael Zock, Michael Carl, Arnt Lykke Jakobsen (Copenhagen Studies in Language 41), Frederiksberg: Samfundslitteratur, 2011; pp.93-104. [PDF, 768KB]

(2011) Jonathan H.Clark, Chris Dyer, Alon Lavie, & Noah A.Smith: Better hypothesis testing for statistical machine translation: controlling for optimizer instability. ACL-HLT 2011: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Short papers, Portland, Oregon, June 19-24, 2011; pp.176-181. [PDF, 146KB]

(2011) Marcello Federico, Luisa Bentivogli, Michael Paul, & Sebastian Stüker: Overview of the IWSLT 2011 evaluation campaign. IWSLT 2011: Proceedings of the International Workshop on Spoken Language Translation, San Francisco, December 8-9, 2011, ed. Marcello Federico, Mei-Yuh Hwang, Margit Rödder, Sebastian Stüker; pp.11-27. [PDF, 486KB]

(2011) Christian Federmann: Results from the ML4HMT shared task on applying machine learning techniques to optimise the division of labour in hybrid MT. [LIHMT] ML4HMT 2011: Shared Task on Applying Machine Learning Techniques to Optimise the Division of Labour in Hybrid Machine Translation, 19th November, Barcelona Media, Barcelona; pp.110-117. [PDF, 695KB]

(2011) Monica Gavrila & Natalia Elita: Comparing corpus-based MT approaches using restricted resources. [LIHMT] International Workshop on Using Linguistic Information for Hybrid Machine Translation, 18th November 2011, Universitat Politècnica de Catalunya, Barcelona; pp.43-49. [PDF, 304KB]

(2011) Monica Gavrila & Natalia Elita: Experiments with small-sized corpora in CBMT. [RANLP 2011] Proceedings of the Student Research Workshop associated with RANLP 2011, Hissar, Bulgaria, 13 September 2011; pp.67-72. [PDF, 113KB]

(2011) Olivier Hamon & Khalid Choukri: Evaluation methodology and results for English-to-Arabic MT.  MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.480-487. [PDF,

(2011) Ea-Ee Jan, Niyu Ge, Shih-Hsiang Lin, & Berlin Chen: An effective and robust framework for transliteration exploration. [IJCNLP 2011] Proceedings of the 5th International Joint Conference on Natural Language Processing, Chiang Mai, Thailand, November 8-13, 2011; pp.1332-1340. [PDF, 204KB]

(2011) Melita Koletnik Korošec: The internet, Google Translate and Google Translator Toolkit – nuisance or necessity in translator training?  Tralogy, Paris, 3-4 March 2011; 17pp. [PDF, 173KB]

(2011) Gorka Labaka, Arantza Díaz de Ilarraza, Cristina España-Bonet, Kepa Sarsola, & Lluís Màrquez: Deep evaluation of hybrid architectures: simple metrics correlated with human judgments. [LIHMT] International Workshop on Using Linguistic Information for Hybrid Machine Translation, 18th November 2011, Universitat Politècnica de Catalunya, Barcelona; pp.50-57. [PDF, 313KB]

(2011) Elena Lloret & Manuel Palomar: Finding the best approach for multi-lingual text summarisation: a comparative analysis. [RANLP 2011] Proceedings of Recent Advances in Natural Language Processing, Hissar, Bulgaria, 12-14 September 2011; pp.194-201. [PDF, 386KB]

(2011) Maria José Machado & Hilário Leal Fontes: Machine translation: case study – English into Portuguese – evaluation of Moses in DGT Portuguese Language Department using Moses for Mere Mortals. A Folha: Boletim da Língua Portuguesa nas Instituições Europeias, no.37, autumn 2011; English version, 11pp. [PDF,

(2011) Johanna Monti, Anabela Barreiro, Annibale Elia, Federica Marano, & Antonella Napoli: Taking on new challenges in multi-word unit processing for machine translation. Proceedings of the Second International Workshop on Free/Open-Source Rule-Based Machine Translation, Barcelona, Spain, January 20-21, 2011, ed. F.Sánchez-Martínez and J.A.Pérez-Ortiz; pp.11-19. [PDF, 148KB]

(2011) Manny Rayner, Paula Estrella, & Pierrette Bouillon: Bootstrapping a statistical speech translator from a rule-based one. Proceedings of the Second International Workshop on Free/Open-Source Rule-Based Machine Translation, Barcelona, Spain, January 20-21, 2011, ed. F.Sánchez-Martínez and J.A.Pérez-Ortiz; pp.21-28. [PDF, 199KB]

 (2011) Johann Roturier & Anthony Bensadoun: Evaluation of MT systems to translate user generated content. MT  Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.244-251. [PDF, 284KB]

(2011) Antonio Toral, Federico Gaspari, Sudip Kumar Naskar, & Andy Way: Comparative evaluation of research vs. online MT systems. [EAMT 2011]: proceedings of the 15th conference of the European Association for Machine Translation, 30-31 May 2011, Leuven, Belgium; eds. Mikel L.Forcada, Heidi Depraetere, Vincent Vandeghinste; pp.13-20. [PDF, 163KB]; presentation, 22 slides [PDF, 173KB]

 (2011) Min Zhang, A.Kumaran, & Haizhou Li: Whitepaper of NEWS 2011 shared task on machine transliteration. [IJCNLP 2011] Proceedings of the 2011 Named Entities Workshop, Chiang Mai, Thailand, November 12, 2011; pp.14-22. [PDF, 240KB]

(2010) Ondřej Bojar, Adam Liška, & Zdeněk Žabokrtský: Evaluating utility of data sources in a large parallel Czech-English corpus CzEng 0.9. LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.447-452. [PDF, 359KB]

(2010) Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, & Omar F.Zaidan: Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation. ACL 2010: Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR. Proceedings of the workshop, 15-16 July 2010, Uppsala University, Uppsala, Sweden; pp. 17-53. [PDF, 837KB]

(2010) Sherri Condon, Dan Parvaz, John Aberdeen, Christy Doran, Andrew Freeman, & Marwan Awad: Evaluation of machine translation errors in English and Iraqi Arabic. LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.729-735. [PDF, 644KB]

(2010) Marta R.Costa-jussà, Mireia Farrús, José B.Mariño, & José A.R.Fonollosa: Automatic and human evaluation study of a rule-based and a statistical Catalan-Spanish machine translation systems. LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.1706-1711. [PDF, 335KB]

(2010) Heidi Depraetere & Pablo Vazquez: Machine translation engine selection in the enterprise. EAMT 2010: Proceedings of the 14th Annual conference of the European Association for Machine Translation, 27-28 May 2010, Saint-Raphaël, France. Proceedings ed.Viggo Hansen and François Yvon; 4pp. [PDF, 480KB]; presentation: 14 slides [PDF, 1631KB]

(2010) Lola García-Santiago, Maria-Dolores Olvera-Lobo: Automatic web translators as part of a multilingual question-answering (QA) system: translation of questions. Translation Journal 14 (1), January 2010; 5pp. [PDF, 117KB]

(2010) Yifan He, Yanjun Ma, Josef van Genabith, & Andy Way: Bridging SMT and TM with translation recommendation. ACL 2010: the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, July 11-16, 2010: Conference proceedings; pp.622-630. [PDF, 573KB]

 (2010) Rafał Jaworski: Computing transfer score in example-based machine translation [abstract]. CICLING 2010: 11th International Conference on Intelligent Text Processing and Computational Linguistics, March 21-27, 2010, Iaşi, Romania; 1p. [PDF, 58KB]

 (2010) Jenny Lu: Machine translation (MT) @ CA Technologies: where can MT be best successful and what are the best MT engines for various languages. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; 3pp. [PDF, 108KB]

(2010) François Masselot, Petra Ribiczey, & Gema Ramírez-Sánchez: Using the Apertium Spanish-Brazilian Portuguese machine translation system for localization. EAMT 2010: Proceedings of the 14th Annual conference of the European Association for Machine Translation, 27-28 May 2010, Saint-Raphaël, France. Proceedings ed.Viggo Hansen and François Yvon; 8pp. [PDF, 577KB]; presentation: 23 slides [PDF, 569KB]

(2010) Mahsa Mohaghegh & Abdolhossein Sarrafzadeh: Performance evaluation of various training data in English-Persian statistical machine translation. JADT 2010: 10th International Conference on Statistical Analysis of Textual Data, 9-11 juin 2010, Rome, Italie; pp.1092-1100. [PDF, 2233KB]

(2010) Sanooch Nathalang: Don’t use big words with me: an evaluation of English-Thai statistical-based machine translation [abstract]. UCCTS 2010: Using Corpora in Contrastive and Translation Studies, Edge Hill University, UK, 27-29 July 2010; p.11. [PDF, 90KB]

(2010) Michael Paul, Marcello Federico, & Sebastian Stüker: Overview of the IWSLT 2010 evaluation campaign. Proceedings of the 7th International Workshop on Spoken Language Translation, 2-3 December 2010, Paris, France; pp.3-27. [PDF, 558KB]

(2010) Diana Santos, Luís Miguel Cabral, Corina Forascu, Pamela Forner, Frederic Gey, Katrin Lamm, Thomas Mandl, Petya Osenova, Anselmo Peñas, Álvaro Rodrigo, Julia Schulz, Yvonne Skalban, & Erik Tjong Kim Sang: GikiCLEF: crosscultural issues in multilingual information access. LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.2346-2353. [PDF, 408KB]

(2010) Barbara Scott & Adriana Beaton: Evaluating vendors for machine translation and post-editing. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; 2pp. [PDF, 30KB]

(2010) Radu Soricut & Abdessamad Echihabi: TrustRank: inducing trust in automatic translations via ranking. ACL 2010: the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, July 11-16, 2010: Conference proceedings; pp.612-621. [PDF, 222KB]

(2010) Lucia Specia, Nicola Cancedda, & Marc Dymetman: A dataset for assessing machine translation evaluation metrics. LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.3375-3378. [PDF, 362KB]

 (2010) Jost Zetzsche: Consider the Luddites. Translation Journal 14 (2) April 2010; 4pp. [PDF, 150KB]

(2010) Li Zuo: Machine translation from English to Chinese: a study of Google’s performance with the UN documents. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; abstract

Examples of MT output

(2012) Michael J.Beaulieu: U.S.Army machine foreign language translation system (MFLTS). AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; abstract, 1p. [PDF, 75KB]; presentation, 10 slides. [PDF of PPT, 1081KB]

(2012) Alan K.Melby, Jason Housley, Paul J.Fields, & Emily Tuioti: Reliably assessing the quality of post-edited translation based on formalized structured translation specifications. AMTA-2012: Workshop on post-editing technology and practice. Proceedings, San Diego, October 28, 2012; 10pp. [PDF, 181KB]

(2011) Omar F.Zaidan & Chris Callison-Burch: Crowdsourcing translation: professional quality from non-professionals. ACL-HLT 2011: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, Portland, Oregon, June 19-24, 2011; pp.1220-1229. [PDF, 241KB]

(2010) Danielle Belopotosky: Putting Google to the test. New York Times, March 9, 2010. [PDF, 169KB]

Fluency see Readability and comprehension

Human-targeted Translation Edit Rate (HTER)

(2014) Eleftherios Avramidis: Efforts on machine learning over human-mediated translation edit rate. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.302-306. [PDF, 331KB]

(2014) Isabel Lacruz, Michael Denkowski, & Alon Lavie: Cognitive demand and cognitive effort in post-editing. AMTA 2014: proceedings of the eleventh conference of the Association for Machine Translation in the Americas, Vancouver, BC, October 22-26; Workshop on Post-editing Technology and Practice (WPTP-3); pp.73-84. [PDF, 126KB]

(2013) José Guilherme Camargo de Souza, Christian Buck, Marco Turchi & Matteo Negri: FBK-UEdin participation to the WMT13 quality estimation shared task.  WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.352-358. [PDF, 190KB]

(2012) Michael Denkowski & Alon Lavie: Challenges in predicting machine translation utility for human post-editors. AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 10pp. [PDF, 166KB]

 (2011) Chi-kiu Lo & Dekai Wu: Structured vs. flat semantic role representations for machine translation evaluation. Proceedings of SSST-5, Fifth Workshop on Syntax, Semantics and Structure in Statistical Translation, ACL HLT 2011, Portland, Oregon, USA, June 2011; pp.10-20. [PDF, 758KB]

(2010) Sherri Condon, Dan Parvaz, John Aberdeen, Christy Doran, Andrew Freeman, & Marwan Awad: Evaluation of machine translation errors in English and Iraqi Arabic. LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.729-735. [PDF, 644KB]

(2010) Michael Denkowski & Alon Lavie: Extending the METEOR machine translation evaluation metric to the phrase level. NAACL HLT 2010: Human Language Technologies: the 2010 annual conference of the North American Chapter of the Association for Computational Linguistics. Proceedings… June 2-4, 2010, Los Angeles, California; pp.250-253. [PDF, 99KB]

(2010) Lucia Specia & Atefeh Farzindar: Estimating machine translation post-editing effort with HTER. JEC 2010: Second joint EM+/CNGL WorkshopBringing MT to the user: research on integrating MT in the translation industry”, AMTA 2010, Denver, Colorado, November 4, 2010; pp.33-41. [PDF, 178KB]

(2010) Omar F.Zaidan & Chris Callison-Burch: Predicting human-targeted translation edit rate via untrained human annotators. NAACL HLT 2010: Human Language Technologies: the 2010 annual conference of the North American Chapter of the Association for Computational Linguistics. Proceedings… June 2-4, 2010, Los Angeles, California; pp.369-372. [PDF, 206KB]

Informativeness

(2012) Marco Turchi, Josef Steinberger, & Lucia Specia: Relevance ranking for translated texts. EAMT 2012: Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, May 28-30 2012, ed. Mauro Cettolo, Marcello Federico, Lucia Specia, Andy Way; pp.153-160. [PDF, 228KB]

Minimum error rate training [MERT]

(2014) Quoc Kanh Do, Teresa Herrmann, Jan Niehues, Alexandre Allauzen, François Yvon & Alex Waibel: The KIT-LIMSI translation system for WMT 2014. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.84-89. [PDF, 378KB]

(2013) Michel Galley, Chris Quirk, Colin Cherry, & Kristina Toutanova: Regularized minimum error rate training. [EMNLP 2013] Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Washington, USA, 18-21 October 2013; pp.1948-1959. [PDF, 207KB]

(2013) Shahab Jalalvand & Daniele Falavigna: Parameter optimization for iterative confusion network decoding in weather-domain speech recognition. [IWSLT 2013] Proceedings of the 10th International Workshop on Spoken Language Translation, Heidelberg, Germany, Dec.5-6, 2013; 5pp. [PDF, 631KB]

(2012) Tagyoung Chung & Michel Galley: Direct error rate minimization for statistical machine translation. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.468-479. [PDF, 673KB]

(2012) Chris Dyer: Discriminative training of language models. Machine Translation Marathon 2012 September 3-8, Edinburgh, UK; 83 slides [PDF of PPT, 1322KB]

(2012) Cristina España-Bonet, Gorka Labaka, Arantza Díaz de Ilarranza, Lluís Màrquez, & Kepa Sarasola: Deep evaluation of hybrid architectures: use of different metrics in MERT weight optimization. In: Free/Open-Source Rule-Based Machine Translation, ed.Cristina España-Bonet and Aarne  Ranta. Proceedings of a Workshop held in Gothenburg, 14-15 June, 2012; pp.65-76. [PDF, 498KB]

(2012) Meritxell Gonzàlez & Cristina España-Bonet: New development functionality for the Asiya Suite parameter optimization with Mert. Machine Translation Marathon 2012 September 3-8, Edinburgh, UK; 9 slides [PDF of PPT, 474KB]

(2012) Sabine Hunsicker, Chen Yu, & Christian Federmann: Machine learning for hybrid machine translation. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.312-316. [PDF, 131KB]

(2012) Lemao Liu, Tiejun Zhao, Taro Watanabe, Hailong Cao, & Conghui Zhu: Expected error minimization with ultraconservative update for SMT. Proceedings of COLING 2012: Posters, Mumbai, December 2012; pp.723-732. [PDF, 144KB]

(2012) Thomas Meyer, Andrei Popescu-Belis, Najeh Hajlaoui, & Andrea Gesmundo: Machine translation of labeled discourse connectives. AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 10pp. [PDF, 196KB]

(2012) Antonio Toral, Sudip Kumar Naskar, Federico Gaspari, & Declan Groves: DELiC4MT: a tool for diagnostic MT evaluation over user-defined linguistic phenomena. Prague Bulletin of Mathematical Linguistics 98, October 2012; pp.121-131. [PDF, 156KB]

(2012) Aurelien Waite, Graeme Blackwood, & William Byrne: Lattice-based minimum error rate training using weighted finite-state transducers with tropical polynomial weights. FSMNLP 2012: Proceedings of the 10th International Workshop on Finite State Methods and Natural Language Processing, July 23-25, 2012, University of the Basque Country, Donostia-San Sebastián; pp.116-125. [PDF, 277KB]

(2011) Alexandra Birch & Miles Osborne: Reordering metrics for MT. ACL-HLT 2011: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, Portland, Oregon, June 19-24, 2011; pp.1027-1035. [PDF, 976KB]

(2011) Mauro Cettolo, Nicola Bertoldi, & Marcello Federico: Methods for smoothing the optimizer instability in SMT. MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.32-39. [PDF, 150KB]

(2011) Marta R.Costa-jussà & Rafael E.Banchs: The BM-I2R Haitian-Créole-to-English translation system description for the WMT 2011 evaluation campaign. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.452-456. [PDF, 101KB]

(2011) Michel Galley & Chris Quirk: Optimal search for minimum error rate training. [EMNLP 2011] Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Edinburgh, Scotland, UK, July 27-31, 2011; pp.38-49. [PDF, 1280KB]

(2011) Barry Haddow, Abhishek Arun, & Philipp Koehn: SampleRank training for phrase-based machine translation. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.261-271. [PDF, 194KB]

(2011) Shujie Liu, Chi-Ho Li, & Ming Zhou: A unified SMT framework combining MIRA and MERT. MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.181-188. [PDF, 233KB]

(2011) Christophe Servan & Holger Schwenk: Optimising multiple metrics with MERT. Sixth Machine Translation Marathon, 5-10 September 2011, Trento; Prague Bulletin of Mathematical Linguistics, no.96, October 2011; pp.109-117. [PDF, 141KB]; presentation, 20 slides [PDF of PPT, 535KB]

(2011) Patrick Simianer, Katharina Wäschle, & Stefan Riezler: Multi-task minimum error rate training for SMT. Sixth Machine Translation Marathon, 5-10 September 2011, Trento; Prague Bulletin of Mathematical Linguistics, no.96, October 2011; pp.99-108. [PDF, 138KB]; presentation, 45 slides [PDF of PPT, 295KB]

(2011) Artem Sokolov & François Yvon: Minimum error rate training semiring. [EAMT 2011]: proceedings of the 15th conference of the European Association for Machine Translation, 30-31 May 2011, Leuven, Belgium; eds. Mikel L.Forcada, Heidi Depraetere, Vincent Vandeghinste; pp.241-248. [PDF, 418KB]; presentation, 19 slides [PDF]

 (2011) Jun Suzuki, Kevin Duh, & Masaaki Nagata: Distributed minimum error rate training of SMT using particle swarm optimization. [IJCNLP 2011] Proceedings of the 5th International Joint Conference on Natural Language Processing, Chiang Mai, Thailand, November 8-13, 2011; pp.649-657. [PDF, 1101KB]

(2011) Yinggong Zhao, Shujie Liu, Yangsheng Ji, Jiajun Chen, & Guodong Zhou: Transductive minimum error rate training for statistical machine translation. [IJCNLP 2011] Proceedings of the 5th International Joint Conference on Natural Language Processing, Chiang Mai, Thailand, November 8-13, 2011; pp.641-648. [PDF, 214KB]

(2010) Ralf D.Brown: Taming structured perceptrons on wild feature vectors. ACL 2010: Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR. Proceedings of the workshop, 15-16 July 2010, Uppsala University, Uppsala, Sweden; pp. 384-391. [PDF, 132KB]

(2010) Daniel Cer, Christopher D.Manning, & Daniel Jurafsky: The best lexical metric for phrase-based statistical MT system optimization. NAACL HLT 2010: Human Language Technologies: the 2010 annual conference of the North American Chapter of the Association for Computational Linguistics. Proceedings… June 2-4, 2010, Los Angeles, California; pp.555-563. [PDF, 169KB]

 (2010) Samidh Chatterjee & Nicola Cancedda: Minimum error rate training by sampling the translation lattice. [EMNLP 2010] Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, MIT, Massachusetts, USA, 9-11 October 2010; pp.606-615. [PDF, 248KB]

(2010) Cristina España-Bonet & Lluís Màrquez: Robust estimation of feature weights in statistical machine translation. EAMT 2010: Proceedings of the 14th Annual conference of the European Association for Machine Translation, 27-28 May 2010, Saint-Raphaël, France. Proceedings ed.Viggo Hansen and François Yvon; 8pp. [PDF, 618KB]

(2010) Samuel Larkin, Boxing Chen, George Foster, Ulrich Germann, Eric Joanis, Howard Johnson, & Roland Kuhn: Lessons from NRC’s Portage system at WMT 2010.  ACL 2010: Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR. Proceedings of the workshop, 15-16 July 2010, Uppsala University, Uppsala, Sweden; pp. 127-132. [PDF, 124KB]

(2010) Shujie Liu, Chi-Ho Li, & Ming Zhou: Discriminative pruning for discriminative ITG alignment. ACL 2010: the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, July 11-16, 2010: Conference proceedings; pp.316-324. [PDF, 972KB]

(2010) Sirvan Yahyaei & Christof Monz: The QMUL system description for IWSLT 2010. Proceedings of the 7th International Workshop on Spoken Language Translation, 2-3 December 2010, Paris, France; pp.157-162. [PDF, 303KB]

Quality assurance

(2015) Klaus Fleischmann: From occasional quality control to collaborative quality assurance. Proceedings of the 37th Conference Translating and the Computer, London, November 26-27, 2015; pp .24-26. [PDF, 141KB]

(2014) Jerzy Czopik: Quality assurance process in translation. Translating and the Computer 36: proceedings. Asling: International Society for Advancement in Language Technology, 27-28 November 2014; pp.77-85. [PDF, 374KB]

(2012) Mikhail Kudinov: New horizons in automation of translation QA. [Aslib 2012] Translating and the Computer 34, 29-30 November 2012, One Birdcage Walk, London, UK; 7pp. [PDF, 383KB]; presentation: 11 slides [PDF, 461KB]

Quality control

(2015) Quinn Lam: Solving specific content challenges with flexible machine translation. MT Summit XV, October 30 – November 3, 2015, Miami, Florida, USA. Proceedings of MT Summit XV: vol.2: MT Users’ Track; pp.32-64. [PDF of PPT, 1534KB] 

(2015) Achim Ruopp: Industry shared metrics with the TAUS dynamic quality dashboard and API.  MT Summit XV, October 30 – November 3, 2015, Miami, Florida, USA. Proceedings of MT Summit XV: vol.2: MT Users’ Track; pp.163-191. [PDF, 2464KB]

(2015) Elaine O’Curran: MT quality evaluations: from test environment to production. MT Summit XV, October 30 – November 3, 2015, Miami, Florida, USA. Proceedings of MT Summit XV: vol.2: MT Users’ Track; pp.106-143. [PDF, 1516KB]

(2015) Paola Valli: The TAUS Quality Dashboard. Proceedings of the 37th Conference Translating and the Computer, London, November 26-27, 2015; pp.127-136. [PDF, 2708KB]

(2014) Laura Casanellas & Lena Marg: Assumptions, expectations and outliers in post-editing. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; pp.93-97. [PDF, 592KB]

(2014) Joanna Drugan: Top-down or bottom-up: what do industry approaches to translation quality mean for effective integration of standards and tools? Translating and the Computer 36: proceedings. Asling: International Society for Advancement in Language Technology, 27-28 November 2014; pp.109-117. [PDF, 327KB]

(2014) Mitesh M.Khapra, Ananthakrishnan Ramanathan, Anoop Kunchukuttan, Karthik Visweswariah, & Pushpak Bhattacharyya: When transliteration met crowdsourcing: an empirical study of transliteration via crowdsourcing using efficient, non-redundant and fair quality control. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.196-202. [PDF, 180KB]

(2013) Anoop Kunchukuttan, Rajen Chatterjee, Shourya Roy, Abhijit Mishra, & Pushpak Bhattacharyya: TransDoop: a map-reduce based crowdsourced translation for complex domains. ACL-2013: Proceedings of the 51st Meeting of the Association for Computational Linguistics, System demonstrations, Sofia, Bulgaria, August 4-9 2013; pp.175-180. [PDF, 1061KB]

(2013) Sergio Penkale & Andy Way: Tailor-made quality-controlled translation. [Aslib 2013] Translating and the Computer 35, 28-29 November 2013, etc.venues, Paddington, London, UK; 7pp. [PDF, 281KB]; presentation, 24 slides [PDF of PPT, 869KB]

(2013) Lori Thicke: The industrial process for quality machine translation. Journal of Specialised Translation, issue 19, January 2013; pp.8-18. [PDF, 303KB]

(2010) Jennifer Beaupre & Kent Taylor: Content quality for better MT: a practical guide to quality at the source. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; 3pp. [PDF, 69KB]

(2010) Daniel Marcu, Kathleen Egan, Chuck Simmons, & Ning-Ning Mahlmann: Utilizing automated translation with quality scores to increase productivity. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; abstract

(2010) Carol Van Ess-Dykema, Jocelyn Phillips, Florence Reeder, & Laurie Gerber: Paralinguist assessment decision factors for machine translation output: a case study. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; 7pp. [PDF, 180KB]; abstract

Quality estimation

(2015) Varvara Logacheva & Lucia Specia: Phrase-level estimation for machine translation. [IWSLT 2015] Proceedings of the International Workshop on Spoken Language Translation, December 3-4, 2015, Da Nang, Vietnam; pp.143-150. [PDF, 3.2MB]

(2015) Elaine O’Curran: MT quality evaluations: from test environment to production. MT Summit XV, October 30 – November 3, 2015, Miami, Florida, USA. Proceedings of MT Summit XV: vol.2: MT Users’ Track; pp.106-143. [PDF, 1516KB]

(2015) Carla Ortiz Boix & Anna Matamala: Quality assessment of postedited versus translated wildlife documentary films: a three-level approach. MT Summit XV, October 30 – November 3, 2015, Miami, Florida, USA. Proceedings of MT Summit XV: Fourth Workshop on Post-editing Technology and Practice (WPTP 4); p.16-30. [PDF, 1089KB]

(2015) Juan Rowda: Machine translation quality estimation: a linguist’s approach. MT Summit XV, October 30 – November 3, 2015, Miami, Florida, USA. Proceedings of MT Summit XV: vol.2: MT Users’ Track; pp.146-162. [PDF, 761KB]

(2015) José G.C.de Souza, Marcello Federico, & Hassan Sawaf: MT quality estimation for e-commerce data. MT Summit XV, October 30 – November 3, 2015, Miami, Florida, USA. Proceedings of MT Summit XV: vol.2: MT Users’ Track; pp.20-29. [PDF, 555KB]  

(2014) Eleftherios Avramidis: Efforts on machine learning over human-mediated translation edit rate. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.302-306. [PDF, 331KB]

(2014) Daniel Beck, Kashif Shah, & Lucia Specia: SHEF-Lite 2.0: sparse multi-task Gaussian processes for translation quality estimation. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.307-312. [PDF, 306KB]

(2014) Ergun Biçici & Andy Way: Referential translation machines for predicting translation quality. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.313-321. [PDF, 378KB]

(2014) Joke Daems, Lieve Macken, & Sonia Vandepitte: On the origin of errors: a fine-grained analysis of MT and PE errors and their relationship. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.62-68. [PDF, 120KB]

(2014) Sandipat Dandapat & Declan Groves: MTWatch: a tool for the analysis of noisy parallel data. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.41-45. [PDF, 190KB]

(2014) José G.C. de Souza, Marco Turchi, Matteo Negri: Towards a combination of online and multitask learning for MT quality estimation: a preliminary study. AMTA 2014: proceedings of the eleventh conference of the Association for Machine Translation in the Americas, Vancouver, BC, October 22-26; Workshop on Interactive and Adaptive Machine Translation; pp.9-19. [PDF, 282KB]

(2014) Irina Galinskaya, Valentin Gusev, Elena Meshcheryakova, & Mariya Shmatova: Measuring the impact of spelling errors on the quality of machine translation. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.2683-2689. [PDF, 448KB]

(2014) A.Görög: Quality evaluation today: the Dynamic Quality Framework. Translating and the Computer 36: proceedings. Asling: International Society for Advancement in Language Technology, 27-28 November 2014; pp.155-164. [PDF, 366KB]

(2014) Chris Hokamp, Iacer Calixto, Joachim Wagner, & Jian Zhang: Target-centric features for translation quality estimation. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.329-334. [PDF, 196KB]

(2014) Sabine Hunsicker & Alexandru Ceausu: Machine translation quality estimation adapted to the translation workflow. Translating and the Computer 36: proceedings. Asling: International Society for Advancement in Language Technology, 27-28 November 2014; pp.133-136. [PDF, 133KB]

(2014) Varvara Logacheva & Lucia Specia: A quality-based active sample selection strategy for statistical machine translation. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.2690-2695. [PDF, 230KB]

(2014) Ngoc-Quang Luong, Laurent Besacier, & Benjamin Lecouteux: LIG system for word level QE task at WMT14. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.335-341. [PDF, 392KB]

(2014) Elaine O’Curran; Translation quality in post-edited versus human-translated  s: a case study. AMTA 2014: proceedings of the eleventh conference of the Association for Machine Translation in the Americas, Vancouver, BC, October 22-26; Workshop on Post-editing Technology and Practice (WPTP-3); pp.113-118. [PDF, 131KB]

(2014) Raphael Rubino, Antonio Toral, Nikola Ljubešić, & Gema Ramírez-Sánchez: Quality estimation for synthetic parallel data generation. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.1843-1849. [PDF, 248KB]

(2014) Carolina Scarton & Lucia Specia: Document-level translation quality estimation: exploring discourse and pseudo-references. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; pp.101-108. [PDF, 409KB]

(2014) Carolina Scarton & Lucia Specia: Exploring consensus in machine translation for quality estimation. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.342-347. [PDF, 148KB]

(2014) Kashif Shah, Marco Turchi, & Lucia Specia: An efficient and user-friendly tool for machine translation quality estimation. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.2560-3564. [PDF, 158KB]

(2014) Kahif Shah & Lucia Specia: Quality estimation for translation selection. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; pp. 109-116. [PDF, 410KB]

(2014) Raivis Skadiņš, Mārcis Pinnis, Andrejs Vasiļjevs, Inguna Skadiņa, & Tomas Hudik: Application of machine translation in localization into low-resourced languages.  Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; pp.209-216. [PDF, 871KB]

(2014) Lucia Specia, & Kashif Shah: Predicting human translation quality. AMTA 2014: proceedings of the eleventh conference of the Association for Machine Translation in the Americas, Vancouver, BC, October 22-26; pp.288-300. [PDF, 717KB]

(2014) Lucia Specia & Kashif Shah: QuEst: a framework for translation quality estimation. AMTA 2014: proceedings of the eleventh conference of the Association for Machine Translation in the Americas, Vancouver, BC, October 22-26; Workshop on Post-editing Technology and Practice (WPTP-3); p.121. [PDF, 81KB]

(2014) José G.C. de Souza, Jesús González-Rubio, Christian Buck, Marco Turchi, & Matteo Negri: FBK-UPV-UEdin participation in the WMT14 quality estimation shared-task. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.322-328. [PDF, 361KB]

(2014) Sara Stymne, Jörg Tiedemann, & Joakim Nivre: Estimating word alignment quality for SMT reordering tasks. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.275-286. [PDF, 344KB]

(2014) Marco Turchi & Bruno Kessler: Automatic annotation of machine translation datasets with binary quality  judgements.  LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.1788-1792. [PDF, 99KB]

(2014) Guillaume Wisniewski, Nicolas Pécheux, Alexandre Allauzen, & François Yvon: LIMSI submission for WMT’14 QE task. [WMT 2014] Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland USA, June 26-27, 2014; pp.348-354. [PDF, 472KB]

(2014) QTLaunchPad: preparation and launch of a large-scale action for quality translation technology. Project duration: July 2012 – June 2014. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; p.133. [PDF, 484KB]

(2013) Hala Almaghout & Lucia Specia: A CCG-based quality estimation metric for statistical machine translation. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.223-230. [PDF, 426KB]

(2013) Eleftherios Avramidis & Maja Popović: Selecting feature sets for comparative and time-oriented quality estimation of machine translation output. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.329-336. [PDF, 196KB]

(2013) Pratyush Banerjee, Raphael Rubino, Johann Roturier, & Josef van Genabith: Quality estimation-guided data selection for domain adaptation of SMT. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.107-108. [PDF, 658KB]

(2013) Daniel Beck, Kashif Shah, Trevor Cohn & Lucia Specia: SHEF-Lite: when less is more for translation quality estimation. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.337-342. [PDF, 330KB]

(2013) Ergun Bicici: Referential translation machines for quality estimation. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.343-351. [PDF, 300KB]

(2013) Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut & Lucia Specia: Findings of the 2013 Workshop on Statistical Machine Translation. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.1-44. [PDF, 1767KB]

(2013) Houda Bouamor, Behrang Mohit, & Kemal Oflazer: SuMT: a framework of summarization and MT. International Joint Conference on Natural Language Processing, Nagoya, Japan, 14-18 October 2013; pp.270-278. [PDF, 154KB]

(2013) José Guilherme Camargo de Souza, Christian Buck, Marco Turchi & Matteo Negri: FBK-UEdin participation to the WMT13 quality estimation shared task.  WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.352-358. [PDF, 190KB]

(2013) Trevor Cohn & Lucia Specia: Modelling annotator bias with multi-task Gaussian processes: an application to machine translation quality estimation. ACL-2013: Proceedings of the 51st Meeting of the Association for Computational Linguistics, Sofia, Bulgaria, August 4-9 2013; pp. 32-42. [PDF, 265KB]

(2013) Joke Daems, Lieve Macken, & Sonia Vandepitte: Quality as the sum of its parts: a two-step approach for the identification of translation problems and translation quality assessment for HT and MT+PE.  Proceedings of MT Summit XIV Workshop on Post-editing Technology and Practice (WPTP-2), Nice, France, 2 September 2013; pp. 63-71. [PDF, 209KB]

(2013) Aswarth Dara, Sandipan Dandapat, Declan Groves, & Josef van Genabith: TMTprime: a recommender system for MT and TM integration. [NAACL-HLT 2013] Proceedings of the NAACL HLT 2013 Demonstration Session, 10-12 June 2013, Atlanta, Georgia; pp.10-13. [PDF, 346KB]

(2013) Stephen Doherty, Declan Groves, Josef van Genabith, Arle Lommel, Aljoscha Burchardt, Hans Uszkoreit, Lucia Specia, & Stelios Piperidis: QT Launch Pad. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; p.439. [PDF, 230KB]

(2013) Mark Fishel: Ranking translations using error analysis and quality estimation. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.405-407. [PDF, 98KB]

(2013) Lluís Formiga, Lluís Màrquez, & Jaume Pujantell: Real-life translation quality estimation for MT system selection. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.69-76. [PDF, 536KB]

(2013) Lluís Formiga, Meritxell Gonzàlez, Alberto Barrón-Cedeño, José A. R. Fonollosa & Lluís Màr­quez: The TALP-UPC approach to system selection: Asiya features and pairwise classification using ran­dom forests. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.359-364. [PDF, 181KB]

(2013) Lluís Formiga, Marta R. Costa-jussà, José B. Mariño, José A. R. Fonollosa, Alberto Barrón-Cedeño & Lluis Marquez: The TALP-UPC phrase-based translation systems for WMT13: system combination with morphology generation, domain adaptation and corpus filtering. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.134-140. [PDF, 159KB]

(2013) Jesús González-Rubio, J.Ramón Navarro-Cerdán & Francisco Casacuberta: Emprical study of a two-step approach to estimate translation quality. [IWSLT 2013] Proceedings of the 10th International Workshop on Spoken Language Translation, Heidelberg, Germany, Dec.5-6, 2013; 8pp. [PDF, 384KB]

(2013) Aaron Li-Feng Han, Yi Lu, Derek F. Wong, Lidia S. Chao, Liangye He & Junwen Xing: Quality estimation for machine translation using the joint method of evaluation criteria and statisti­cal modeling. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.365-372. [PDF, 309KB]

(2013) Almut Silja Hildebrand & Stephan Vogel: MT quality estimation: the CMU system for WMT’13. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.373-379. [PDF, 127KB]

(2013) David Langlois & Kamel Smaili: LORIA system for the WMT13 quality estimation shared task. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.380-385. [PDF, 177KB]

(2013) Arle Richard Lommel, Aljoscha Burchardt, & Hans Uszkoreit: Multidimensional quality metrics: a flexible system for assessing translation quality.  [Aslib 2013] Translating and the Computer 35, 28-29 November 2013, etc.venues, Paddington, London, UK; 7pp. [PDF, 698KB]; presentation, 49 slides [PDF of PPT, 3999KB]

(2013) Ngoc Quang Luong, Benjamin Lecouteux & Laurent Besacier: LIG system for WMT13 QE task: investigating the usefulness of features in word confidence estima­tion for MT. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.386-391. [PDF, 202KB]

(2013) Erwan Moreau & Raphael Rubino: An approach using style classification features for quality estimation. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.429-434. [PDF, 133KB]

(2013) Raphael Rubino, Joachim Wagner, Jennifer Foster, Johann Roturier, Rasoul Samad Zadeh Kaljahi & Fred Hollowood: DCU-Symantec at the WMT 2013 quality estimation shared task. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.392-397. [PDF, 162KB]

(2013) Raphael Rubino, Jennifer Foster, Rasoul Samad Zadeh Kaljahi, Johann Roturier & Fred Hollowood: Estimating the quality of translated user-generated content. International Joint Conference on Natural Language Processing, Nagoya, Japan, 14-18 October 2013; pp.1167-1173. [PDF, 164KB]

(2013) Raphael Rubino, José G.C. de Souza, Jennifer Foster, & Lucia Specia: Topic models for translation quality estimation for gisting purposes. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.295-302. [PDF, 426KB]

(2013) Rasoul Samad Zadeh Kaljahi, Jennifer Foster, Raphael Rubino, Johann Roturier, & Fred Hollowood: Parser accuracy in quality estimation of machine translation: a tree kernel approach. International Joint Conference on Natural Language Processing, Nagoya, Japan, 14-18 October 2013; pp.1092-1096. [PDF, 114KB]

(2013) Kashif Shah, Trevor Cohn, & Lucia Specia: An investigation on the effectiveness of features for translation quality estimation. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.167-174. [PDF, 523KB]

(2013) Anil Kumar Singh, Guillaume Wisniewski & François Yvon: LIMSI submission for the WMT’13 quality estimation task: an experiment with n-gram posteriors. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.398-404. [PDF, 204KB]

(2013) Lucia Specia: Machine translation quality evaluation and estimation [abstract]. [Aslib 2013] Translating and the Computer 35, 28-29 November 2013, etc.venues, Paddington, London, UK; presentation, 80 slides [PDF of PPT, 876KB]

(2013) Lucia Specia, Kashif Shah, Jose G.C.de Souza, & Trevor Cohn: QuEst – a translation quality estimation framework. ACL-2013: Proceedings of the 51st Meeting of the Association for Computational Linguistics, System demonstrations, Sofia, Bulgaria, August 4-9 2013; pp.79-84. [PDF, 163KB]

(2013) Marco Turchi, Matteo Negri & Marcello Federico: Coping with the subjectivity of human judgements in MT quality estimation. WMT 2013: 8th Workshop on Statistical Machine Translation, Proceedings of the Workshop, August 8-9, 2013, Sofia, Bulgaria; pp.240-251. [PDF, 472KB]

(2013) Guillaume Wisniewski, Anil Kumar Singh, Natalia Segal, & François Yvon: Design and analysis of a large corpus of post-edited translations: quality estimation, failure analysis and variability of post-edition. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.117-124. [PDF, 541KB]

(2012) Eleftherios Avramidis: Comparative quality estimation: automatic sentence-level ranking of multiple machine translation outputs. Proceedings of COLING 2012: Technical Papers, Mumbai, December 2012; pp.115-132. [PDF, 235KB]

(2012) Eleftherios Avramidis: Quality estimation for machine translation output using linguistic analysis and decoding features. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.84-90. [PDF, 111KB]

(2012) Nicola Bertoldi, Mauro Cettolo, Marcello Federico, & Christian Buck: Evaluating the learning curve of domain adaptive statistical machine translation systems. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.433-441. [PDF, 141KB]

(2012) Christian Buck: Black box features for the WMT 2012 quality estimation shared task. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.91-95. [PDF, 115KB]

(2012) Mariano Felice & Lucia Specia: Linguistic features for quality estimation.  WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.96-103. [PDF, 257KB]

(2012) Mark Fishel, Rico Sennrich, Maja Popović, & Ondřej Bojar: TerrorCat: a translation error categorization-based MT quality metric. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.64-70. [PDF, 116KB]

(2012) Jesús González Rubio, Alberto Sanchís, & Francisco Casacuberta: PRHLT submission to the WMT12 quality estimation task. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.104-108. [PDF, 111KB]

(2012) Christian Hardmeier, Joakim Nivre, & Jörg Tiedemann: Tree kernels for machine translation quality estimation. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.109-113. [PDF, 423KB]

(2012) David Langlois, Sylvain Raybaud, & Kamel Smaïli: LORIA system for the WMT12 quality estimation shared task. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.114-119. [PDF, 96KB]

(2012) Luong Ngoc Quang: Integrating lexical, syntactic  and system-based features to improve word confidence estimation in SMT. JEP-TALN-RECITAL 2012: Actes de la conference conjointe, Grenoble, 4 au 8 juin 2012; vol.3, pp.43-56. [PDF, 219KB]

(2012) Erwan Moreau & Carl Vogel: Quality estimation: an experimental study using unsupervised similarity measures. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.120-126. [PDF, 139KB]

(2012) Sharon O’Brien: Towards a dynamic quality evaluation model for translation. Journal of Specialised Translation, issue 17, January 2012; pp.55-77. [PDF, 861KB]

(2012) Tsuyoshi Okita, Raphaël Rubino, & Josef van Genabith: Sentence-level quality estimation for MT system combination. COLING 2012: Second Workshop on Applying Machine Learning Techniques to Optimise the Division of Labour in Hybrid MT (ML4HMT, Mumbai, December 2012; pp.55-63. [PDF, 140KB]

(2012) Daniele Pighin, Meritxell González, & Lluís Màrquez: The UPC submission to the WMT 2012 shared task on quality estimation. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.127-132. [PDF, 123KB]

(2012) Maja Popović: Morpheme and POS-based IBM1 scores and language model scores for translation quality estimation.  WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.133-137. [PDF, 110KB]

(2012) Raphael Rubino, Jennifer Foster, Joachim Wagner, Johann Roturier, Rasul Samad Zadeh Kaljahi, & Fred Hollowood: DCU-Symantec submission for the WMT 2012 quality estimation task. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.138-144. [PDF, 125KB]

(2012) Radu Soricut & Sushant Narsale: Combining quality prediction and system selection for improved automatic translation output. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.163-170. [PDF, 135KB]

(2012) Radu Soricut, Nguyen Bach, & Ziyuan Wang: The SDL Language Weaver systems in the WMT12 quality estimation shared task. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.145-151. [PDF, 106KB]

(2012) Lucia Specia: Estimating machine translation quality: state-of-the-art systems and open issues. Machine Translation Marathon 2012 September 3-8, Edinburgh, UK; 127 slides [PDF of PPT, 1667KB]

(2012) Hans Uszkoreit: Quality translation for a multilingual continent: priorities and chances for European MT research [abstract]. JEP-TALN-RECITAL 2012: Actes de la conference conjointe, Grenoble, 4 au 8 juin 2012; vol.4: conferences invités, p.1. [PDF, 70KB]

(2012) Chunyang Wu & Hai Zhao: Regression with phrase indicators for estimating MT quality. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.152-156. [PDF, 119KB]

(2012) Confident MT: estimating translation quality for improved statistical machine translation. [Project paper at] EAMT 2012: Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, May 28-30 2012, ed. Mauro Cettolo, Marcello Federico, Lucia Specia, Andy Way; p.98. [PDF, 424KB]

(2011) Elefterios Avramidis, Maja Popovic, David Vilar, & Aljoscha Burchardt: Evaluate with confidence  estimation: machine ranking of translation outputs using grammatical features. [WMT 2011] Proceedings of the 6th Workshop on Statistical Machine Translation, Edinburgh, Scotland, UK, July 30-31, 2011; pp.65-70. [PDF, 107KB]

(2011) Nguyen Bach, Fei Huang, & Yaser Al-Onaizan: Goodness: a method for measuring machine translation confidence. ACL-HLT 2011: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, Portland, Oregon, June 19-24, 2011; pp.211-219. [PDF, 976KB]

(2011) Christian Hardmeier: Improving machine translation quality prediction with syntactic tree kernels. [EAMT 2011]: proceedings of the 15th conference of the European Association for Machine Translation, 30-31 May 2011, Leuven, Belgium; eds. Mikel L.Forcada, Heidi Depraetere, Vincent Vandeghinste; pp.233-240. [PDF, 633KB] ; presentation, 18 slides [PDF]

 (2011) Lucia Specia: Linguistic indicators for quality estimation of machine translation [abstract]. [LIHMT] International Workshop on Using Linguistic Information for Hybrid Machine Translation, 18th November 2011, Universitat Politècnica de Catalunya, Barcelona; p.14. [PDF, 139KB]

 (2011) Lucia Specia: Quality estimation for machine translation. Proceedings of the Third Joint EM+/CNGL Workshop “Bringing MT to the User: Research Meets Translators” (JEC ’11), Luxembourg, 14 October 2011; 46 slides [PDF of PPT, 749KB]

Quality improvement techniques (see also Interactive methods)

(2015) Quoc Truong Do, Sakriani Sakti, Graham Neubig, Tomoki Toda, & Satoshi Nakamura: Improving translation of emphasis with pause prediction in speech-to-speech translation systems. [IWSLT 2015] Proceedings of the International Workshop on Spoken Language Translation, December 3-4, 2015, Da Nang, Vietnam; pp.204-208. [PDF, 1.7MB]

(2014) Darlene Stewart, Roland Kuhn, Eric Joanis, & George Foster: Coarse “split and lump” bilingual language models for richer source information in SMT. AMTA 2014: proceedings of the eleventh conference of the Association for Machine Translation in the Americas, Vancouver, BC, October 22-26; pp.28-41. [PDF, 1190KB]

 (2013) Yuki Arase & Ming Zhou: Machine translation detection from monolingual web-text.  ACL-2013: Proceedings of the 51st Meeting of the Association for Computational Linguistics, Sofia, Bulgaria, August 4-9 2013; pp.1597-1607. [PDF, 591KB]

(2013) Stephen Doherty, Declan Groves, Josef van Genabith, Arle Lommel, Aljoscha Burchardt, Hans Uszkoreit, Lucia Specia, & Stelios Piperidis: QT Launch Pad. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; p.439. [PDF, 230KB]

 (2013) Meritxell Gonzàlez, Laura Mascarell, & Lluís Màrquez: tSearch: flexible and fast search over automatic translations for improved quality/error analysis.  ACL-2013: Proceedings of the 51st Meeting of the Association for Computational Linguistics, System demonstrations, Sofia, Bulgaria, August 4-9 2013; pp.181-186. [PDF, 963KB]

(2013) Lori Thicke: The industrial process for quality machine translation. Journal of Specialised Translation, issue 19, January 2013; pp.8-18. [PDF, 303KB]

(2011) Christian Hardmeier: Improving machine translation quality prediction with syntactic tree kernels. [EAMT 2011]: proceedings of the 15th conference of the European Association for Machine Translation, 30-31 May 2011, Leuven, Belgium; eds. Mikel L.Forcada, Heidi Depraetere, Vincent Vandeghinste; pp.233-240. [PDF, 633KB] ; presentation, 18 slides [PDF]

 (2011) Els Lefever & Véronique Hoste: An evaluation and possible improvement path for current SMT behavior on ambiguous nouns. Proceedings of SSST-5, Fifth Workshop on Syntax, Semantics and Structure in Statistical Translation, ACL HLT 2011, Portland, Oregon, USA, June 2011; pp.52-60. [PDF, 164KB]

(2011) Omar F.Zaidan & Chris Callison-Burch: Crowdsourcing translation: professional quality from non-professionals. ACL-HLT 2011: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, Portland, Oregon, June 19-24, 2011; pp.1220-1229. [PDF, 241KB]

Quality measures see Evaluation measures and metrics

Reading and comprehension

(2014) Victoria Porro, Johanna Gerlach, Pierrette Bouillon, & Violeta Seretan: Rule-based automatic post-processing of SMT output to reduce human post-editing effort. Translating and the Computer 36: proceedings. Asling: International Society for Advancement in Language Technology, 27-28 November 2014; pp.66-76. [PDF, 447KB]

(2013) Maxine Eskenazi, Yibin Lin, & Oscar Saz: Tools for non-native readers: the case for translation and simplification. [NAACL HLT 2013] Proceedings of the 2nd Workshop of Natural Language Processing for Improving Textual Accessibility (NLP4ITA), Atlanta, Georgia, 14 June, 2013; pp.20-28. [PDF, 297KB]

(2013) Abhijit Mishra, Pushpak Bhattacharyya, & Michael Carl: Automarically predicting sentence translation difficulty.  ACL-2013: Proceedings of the 51st Meeting of the Association for Computational Linguistics, Short papers, Sofia, Bulgaria, August 4-9 2013; pp.346-351. [PDF, 430KB]

(2012) Michael Carl & Barbara Dragsted: Inside the monitor model: processes of default and challenged translation production. Translation: Corpora, Computation, Cognition 2 (1), July 2012; pp.127-145. [PDF, 2189KB]

(2012) Stephen Doherty: Investigating the effects of controlled language on the reading and comprehension of machine translated texts: a mixed-methods approach. Thesis sumitted for the degree of Doctor of Philosophy, … Dublin City University, January 2012; xiii, 275pp. and appendices. [PDF, 3456KB]

(2012) Anthony Hartley, Midori Tatsumi, Hitoshi Isahara, Kyo Kageura, & Rei Miyata: Readability and translatability judgments for ‘Controlled Japanese’. EAMT 2012: Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, May 28-30 2012, ed. Mauro Cettolo, Marcello Federico, Lucia Specia, Andy Way; pp.237-244 [PDF, 465KB]

(2012) Tucker Maney, Linda Sibert, Dennis Perzanowski, Kalyan Gupta, & Astrid Schmidt-Nielsen: Toward determining the comprehensibility of machine translations. NAACL-HLT Workshop on Predicting and Improving Text Readability for target reader populations (PITR 2012), Montréal, Canada, June 7, 2012; pp.1-7. [PDF, 318KB]

(2012) Sara Stymne, Henrik Danielsson, Sofia Bremin, Hongzhan Hu, Johanna Karlsson, Anna Prytz Lillkull, & Martin Wester: Eye tracking as a tool for machine translation error analysis.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.1121-1126. [PDF, 357KB]

(2012) Midori Tatsumi, Anthony Hartley, Hitoshi Isahara, Kyo Kageura, Toshio Okamoto, & Katsumasa Shimizu: Building translation awareness in occasional authors:a user case from Japan. EAMT 2012: Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, May 28-30 2012, ed. Mauro Cettolo, Marcello Federico, Lucia Specia, Andy Way; pp.53-56. [PDF, 264KB]

(2012) Sandra Weiss & Lars Ahrenberg: Error profiling for evaluation of machine-translated text: a Polish-English case study.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.1764-1770. [PDF, 386KB]

(2012) Sander Wubben, Antal van den Bosch, & Emiel Krahmer: Sentence simplification by monolingual machine translation. [ACL 2012] Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, Jeju, Republic of Korea, 8-14 July 2012; pp.1015-1024. [PDF, 139KB]

(2010) Graeme Blackwood, Adrià de Gispert & William Byrne: Fluency constraints for minimum Bayes-risk decoding of statistical machine translation lattices. Coling 2010: 23rd International Conference on Computational Linguistics. Proceedings of the conference, 23-27 August 2010, Beijing International Convention Center, Beijing, China; pp.71-79. [PDF, 547KB]

(2010) Zhemin Zhu, Delphine Bernhard, & Iryna Gurevych: A monolingual tree-based translation model for sentence simplification. Coling 2010: 23rd International Conference on Computational Linguistics. Proceedings of the conference, 23-27 August 2010, Beijing International Convention Center, Beijing, China; pp.1353-1361. [PDF, 260KB]

Reference

(2012) Sandra Weiss & Lars Ahrenberg: Error profiling for evaluation of machine-translated text: a Polish-English case study.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.1764-1770. [PDF, 386KB]

Reference translations see Evaluation measures and metrics

Relevance ranking

(2012) Marco Turchi, Josef Steinberger, & Lucia Specia: Relevance ranking for translated texts. EAMT 2012: Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, May 28-30 2012, ed. Mauro Cettolo, Marcello Federico, Lucia Specia, Andy Way; pp.153-160. [PDF, 228KB]

Round-trip translation see Back translation

Simplification see Reading and comprehension

Test corpora

(2014) Zdeňka Urešová, Ondřej Dušek, Jan Hajič, & Pavel Pecina: Multilingual test sets for machine translation of search queries for cross-lingual information retrieval in the medical domain. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.3244-3247. [PDF, 116KB]

(2010) Michael Bloodgood & Chris Callison-Burch: Using Mechanical Turk to build machine translation evaluation sets. Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, Los Angeles, CA, June 2010; pp.208-211. [PDF, 89KB]

(2010) Lucia Specia, Nicola Cancedda, & Marc Dymetman: A dataset for assessing machine translation evaluation metrics. LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.3375-3378. [PDF, 362KB]

Translatability

(2015) Rei Miyata, Anthony Hartley, Cécile Paris, Midori Tatsumi, & Kyo Kageura: Japanese controlled language rules to improve machine translatability of municipal documents. MT Summit XV, October 30 – November 3, 2015, Miami, Florida, USA. Proceedings of MT Summit XV: vol.1: MT Researchers’ Track; pp.90-103. [PDF, 466KB]

(2014) Antonio Toral & Andy Way: Is machine translation ready for literature? Translating and the Computer 36: proceedings. Asling: International Society for Advancement in Language Technology, 27-28 November 2014; pp.174-176. [PDF, 103KB]

(2013) Shachar Mirkin, Sriram Venkatapathy, Marc Dymetman, & Ioan Calapodescu: SORT: an interactive source-rewriting tool for improved translation. ACL-2013: Proceedings of the 51st Meeting of the Association for Computational Linguistics, System demonstrations, Sofia, Bulgaria, August 4-9 2013; pp.85-90. [PDF, 943KB]

(2013) Abhijit Mishra, Pushpak Bhattacharyya, & Michael Carl: Automarically predicting sentence translation difficulty.  ACL-2013: Proceedings of the 51st Meeting of the Association for Computational Linguistics, Short papers, Sofia, Bulgaria, August 4-9 2013; pp.346-351. [PDF, 430KB]

(2012) Anthony Hartley, Midori Tatsumi, Hitoshi Isahara, Kyo Kageura, & Rei Miyata: Readability and translatability judgments for ‘Controlled Japanese’. EAMT 2012: Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, May 28-30 2012, ed. Mauro Cettolo, Marcello Federico, Lucia Specia, Andy Way; pp.237-244 [PDF, 465KB]

(2012) Martha Thunes: An analysis of translational complexity in two text types. Translation: Corpora, Computation, Cognition 2 (1), July 2012; pp.81-101. [PDF, 506KB]

(2010) Midori Tatsumi & Johann Roturier: Source text characteristics and technical and temporal post-editing effort: what is their relationship? JEC 2010: Second joint EM+/CNGL WorkshopBringing MT to the user: research on integrating MT in the translation industry”, AMTA 2010, Denver, Colorado, November 4, 2010; pp.43-51. [PDF, 10,492KB]

(2010) Carol Van Ess-Dykema, Jocelyn Phillips, Florence Reeder, & Laurie Gerber: Paralinguist assessment decision factors for machine translation output: a case study. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; 7pp. [PDF, 180KB]; abstract

Translation Edit Rate [TER] (see also Human-targeted Edit Rate)

(2014) Raphael Rubino, Antonio Toral, Nikola Ljubešić, & Gema Ramírez-Sánchez: Quality estimation for synthetic parallel data generation. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.1843-1849. [PDF, 248KB]

(2013) Lisa N.Michaud & Patricia Ann McCoy: Applying machine translation metrics to student-written translations. [NAACL HLT 2013] Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, Atlanta, Georgia, June 13 2013; pp.306-311. [PDF, 481KB]

(2011) Houda Bouamor, Aurélien Max, & Anne Vilnat: Monolingual alignment by edit rate computation on sentential paraphrase pairs. ACL-HLT 2011: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Short papers, Portland, Oregon, June 19-24, 2011; pp.395-400. [PDF, 138KB]

(2011) Wei-Yun Ma & Kathleen McKeown: System combination for machine translation based on text-to-text generation. MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.546-553. [PDF,

 (2010) Sherri Condon, Dan Parvaz, John Aberdeen, Christy Doran, Andrew Freeman, & Marwan Awad: Evaluation of machine translation errors in English and Iraqi Arabic. LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.729-735. [PDF, 644KB]

(2010) Jinhua Du & Andy Way: Using TERp to augment the system combination for SMT. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; 9pp. [PDF, 177KB]

Translationese

(2014) Joaquim Moré & Salvador Climent: Machine translationness: machine-likeness in machine translation evaluation. LREC 2014: Ninth International Conference on Language Resources and Evaluation, May 26-31, 2014 Harpa Concert Hall and Conference Center, Reykjavik, Iceland; ed. Nicoletta Calzolari et al.; pp.54-61. [PDF, 464KB]

(2013) Yuki Arase & Ming Zhou: Machine translation detection from monolingual web-text.  ACL-2013: Proceedings of the 51st Meeting of the Association for Computational Linguistics, Sofia, Bulgaria, August 4-9 2013; pp.1597-1607. [PDF, 591KB]

(2013) Gennadi Lembersky, Noam Ordan, & Shuly Wintner: Improving statistical machine translation by adapting translation models to translationese. Computational Linguistics 39 (4); pp.999-1023. [PDF, 220KB]

(2013) Sergiu Nisioi & Liviu P.Dinu: A clustering approach for translationese identification. Proceedings of Recent Advances in Natural  Language Processing, Hissar, Bulgaria, 7-13 September 2013; pp.532-538. [PDF, 489KB]

(2012) Gennadi Lembersky, Noam Ordan, & Shuly Wintner: Adapting translation models to translationese improves SMT. [EACL 2012] Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, Avignon, France, April 23-27, 2012; pp.255-265. [PDF, 173KB]

(2011) Moshe Koppel & Noam Ordan: Translationese and its dialects. ACL-HLT 2011: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, Portland, Oregon, June 19-24, 2011; pp.1318-1326. [PDF, 244KB]

(2011) Gennadi Lembersky, Noam Ordan, & Shuly Wintner: Language models for machine translation: original vs. translated texts. [EMNLP 2011] Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Edinburgh, Scotland, UK, July 27-31, 2011; pp.363-374. [PDF, 288KB]

(2011) Gennadi Lembersky, Noam Ordan, & Shuly Wintner: Language models for machine translation: original vs. translated texts. Machine Translation and Morphologically- rich Languages: Research Workshop of the Israel Science Foundation, University of Haifa, Israel, 26 January, 2011; 2pp. [PDF, 112KB]; presentation: 40 slides [PDF of PPT, 382KB]

(2011) Marius Popescu: Studying translationese at the character level.  [RANLP 2011] Proceedings of Recent Advances in Natural Language Processing, Hissar, Bulgaria, 12-14 September 2011; pp.634-639. [PDF, 121KB]

(2011) Spencer Rarrick, Chris Quirk, & Will Lewis: MT detection in web-scraped parallel corpora. MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.422-429. [PDF, 323KB]

Usability of systems

(2014) Sheila Castilho, Sharon O’Brien, Fabio Alves, & Morgan O’Brien: Does post-editing increase usability? A study with Brazilian Portuguese as target language. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; pp.183-190. [PDF, 754KB]

(2013) Alon Lavie, Olga Beregovaya, Michael Denkowski, & David Clarke: Analyzing and predicting MT utility and post-editing productivity in enterprise-scale translation projects. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.305-307. [PDF, 258KB]

(2012) Sergey Block, Michael Bloodgood, Petra Bradley, Ryan Corbett, Michael Maxwell, Erica Michael, Peter Osthus, Paul Rodrigues, & Benjamin Strauss: Evaluating parallel corpora: assessing utility for use with translation memory systems in government settings [abstract]. AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 1p. [PDF, 12KB]; presentation, 39 slides [PDF of PPT, 1589KB]

(2012) Sachiyo Demizu & Mike Holland: Linguists love art and management loves efficiency – can MT be the solution? AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 10pp. [PDF, 106KB]

(2012) Michael Denkowski & Alon Lavie: Challenges in predicting machine translation utility for human post-editors. AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 10pp. [PDF, 166KB]

(2012) Stephen Doherty & Sharon O’Brien: A user-based usability assessment of raw machine translated technical instructions. AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 6pp. [PDF, 143KB]

(2011) Mark Seligman & Mike Dillinger: Real-time multi-media translation for healthcare: a usability study. MT Summit XIII: the Thirteenth Machine Translation Summit [organized by the] Asia-Pacific Association for Machine Translation (AAMT), 19-23 September 2011, Xiamen, China; pp.595-602. [PDF, 574KB]

(2010) Heidi Depraetere & Pablo Vazquez: Machine translation engine selection in the enterprise. EAMT 2010: Proceedings of the 14th Annual conference of the European Association for Machine Translation, 27-28 May 2010, Saint-Raphaël, France. Proceedings ed.Viggo Hansen and François Yvon; 4pp. [PDF, 480KB]; presentation: 14 slides [PDF, 1631KB]

(2010) Chi-kiu Lo & Dekai Wu: Evaluating machine translation utility via semantic role labels. LREC 2010: proceedings of the  seventh international conference on Language Resources and Evaluation, 17-23 May 2010, Valletta, Malta; pp.2873-2877. [PDF, 417KB]

User evaluation and feedback

(2015) Jasmine Ang, Marc Randell Chan, John Paolo Genato, Joyce Uy, & Joel Ilao: Development of a Filipino-to-English bidirectional statistical machine translation system that dynamically updates via user feedback. [IWSLT 2015] Proceedings of the International Workshop on Spoken Language Translation, December 3-4, 2015, Da Nang, Vietnam; pp.225-231. [PDF, 2.3MB]

(2015) Miguel Á.Candel-Mora: Evaluation of English to Spanish MT output of Tourism 2.0 consumer-generated reviews with post-editing purposes. Proceedings of the 37th Conference Translating and the Computer, London, November 26-27, 2015; pp.37-47. [PDF, 284KB]

(2014) Linda Mitchell, Johann Roturier, & David Silva: Using the ACCEPT framework to conduct an online community-based translation evaluation study. Proceedings of the 17th annual conference of the European Association for Machine Translation, EAMT 2014, Dubrovnik, Croatia, 16th-18th June 2014, edited by Marko Tadić, Philipp Koehn, Johann Roturier, Andy Way; pp.155-158. [PDF, 361KB]

 (2013) V.Alabau, J.González-Rubio, L.A.Leiva, D.Ortiz-Martínez, G.Sanchis-Trilles, F.Casacuberta, B.Meso-Lao, R.Bonk. M.Carl, & M.García-Martínez: User evaluation of advanced interaction features for a computer-assisted translation workbench. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.361-368. [PDF, 1166KB]

(2013) Laura Bota, Christoph Schneider, & Andy Way: COACH: designing a new CAT tool with translator interaction. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.313-320. [PDF, 383KB]

(2013) Pierrette Bouillon: Automated Community Content Editing PorTal (ACCEPT). Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; p.407. [PDF, 182KB]

(2013) William Byrne & Lluís Màrquez: FAUST: Feedback Analysis for User Adaptive Statistical Translation. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; p.421. [PDF, 635KB]

(2013) Laura Casanellas & Lena Marg: Connectivity, adaptability, productivity, quality, price: getting the MT recipe right. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp. 371-379. [PDF of PPT, 2615KB]

(2013) Mauro Cettolo, Christophe Servan, Nicola Bertoldi, Marcello Federico, Loïc Barrault, & Holger Schwenk: Issues in incremental adaptation of statistical MT from human post-edits. Proceedings of MT Summit XIV Workshop on Post-editing Technology and Practice (WPTP-2), Nice, France, 2 September 2013; pp. 111-118. [PDF, 186KB]

(2013) Mauro Cettolo, Nicola Bertoldi, & Marcello Federico: Project adaptation for MT-enhanced computer assisted translation. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.27-34. [PDF, 399KB]

(2013) An-Chang Hsieh, Hen-Hsen Huang, & Hsin-His Chen: Uses of monolingual in-domain corpora for cross-domain adaptation with hybrid MT approaches. Proceedings of the Second Workshop on Hybrid Approaches to Translation, Sofia, Bulgaria, August 8, 2013; pp.117-122. [PDF, 275KB]

(2013) Joss Moorkens & Sharon O’Brien: User attitudes to the post-editing interface.  Proceedings of MT Summit XIV Workshop on Post-editing Technology and Practice (WPTP-2), Nice, France, 2 September 2013; pp. 19-25. [PDF, 268KB]

(2013) Maja Popović, Eleftherios Avramidis, Aljoscha Burchardt, Sabine Hunsicker, Sven Schmeier, Cindy Tscherwinka, David Vilar, & Hans Uszkoreit: Learning from human judgments of machine translation output. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.231-238. [PDF, 328KB]

(2013) Akiko Sakamoto, Kazuhiko Abe, Kazuo Sumita & Satoshi Kamatani: Evaluation of a simultaneous interpretation system and analysis of speech log for user experience assessment. [IWSLT 2013] Proceedings of the 10th International Workshop on Spoken Language Translation, Heidelberg, Germany, Dec.5-6, 2013; 8pp. [PDF, 691KB]

(2013) K.Wäschle, P.Simianer, N.Bertoldi, S.Riezler, & M.Federico: Generative and discriminative methods for online adaptation in SMT. Proceedings of the XIV Machine Translation Summit, Nice, September 2-6, 2013; ed. K.Sima’an, M.L.Forcada, D.Grasmick, H.Depraetere, A.Way; pp.11-18. [PDF, 420KB]

(2012) Vicent Alabau, Luis A. Leiva, Daniel OrtizMartínez, & Francisco Casacuberta: User evaluation of interactive machine translation systems. EAMT 2012: Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, May 28-30 2012, ed. Mauro Cettolo, Marcello Federico, Lucia Specia, Andy Way; pp.20-23. [PDF, 411KB]

(2012) Eleftherios Avramidis, Aljoscha Burchardt, Christian Federmann, Maja Popović, Cindy Tscherwinka, & David  Vilar: Involving language professionals in the evaluation of machine translation. LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.1127-1130. [PDF, 320KB]]

(2012) Michael J.Beaulieu: U.S.Army machine foreign language translation system (MFLTS). AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; abstract, 1p. [PDF, 75KB]; presentation, 10 slides. [PDF of PPT, 1081KB]

(2012) Frédéric Blain, Holger Schwenk, & Jean Senellart: Incremental adaptation using translation information and post-editing analysis. IWSLT-2012: 9th International Workshop on Spoken Language Translation, Hong Kong, December 6th-7th, 2012; pp.229-236. [PDF, 873KB]

(2012) Sachiyo Demizu & Mike Holland: Linguists love art and management loves efficiency – can MT be the solution? AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 10pp. [PDF, 106KB]

(2012) Catherine Dove, Olga Loskutova, & Ruben de la Fuente: What’s your pick: RbMT, SMT or hybrid? AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 3pp. [PDF, 86KB]

(2012) Jie Jiang, Andy Way, & Rejwanul Haque: Translating user-generated content in the social networking space. AMTA-2012: the Tenth Biennial Conference of the Association for Machine Translation in the Americas. Proceedings, San Diego, CA, October 28 – November 1, 2012; 9pp. [PDF, 130KB]

(2012) Philipp Koehn: Computer-aided translation. Machine Translation Marathon 2012 September 3-8, Edinburgh, UK; 118 slides [PDF of PPT, 958KB]

(2012) Chi-kiu Lo, Anand Karthik Tumuluru, & Dekai Wu: Fully automatic semantic MT evaluation.  WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.243-252. [PDF, 379KB]

(2012) Yashar Mehdad, Matteo Negri, & Marcello Federico: Match without a referee: evaluating MT adequacy without reference translations. WMT 2012: 7th Workshop on Statistical Machine Translation. Proceedings of the workshop, June 7-8, 2012, Montréal, Canada; pp.171-180. [PDF, 126KB]

(2012) Linda Mitchell & Johann Roturier: Evaluation of machine-translated user generated content: a pilot study based on user ratings. EAMT 2012: Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, May 28-30 2012, ed. Mauro Cettolo, Marcello Federico, Lucia Specia, Andy Way; pp.61-64. [PDF, 161KB]

(2012) Michael Paul, Eiichiro Sumita, Luisa Bentivogli, & Marcello Federico: Crowd-based MT evaluation for non-English target languages. EAMT 2012: Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, May 28-30 2012, ed. Mauro Cettolo, Marcello Federico, Lucia Specia, Andy Way; pp.229-237. [PDF, 260KB]

(2012) Daniele Pighin, Lluís Màrquez, & Jonathan May: An analysis (and an annotated corpus) of user responses to machine translation output.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.1131-1135. [PDF, 355KB]

(2012) Daniele Pighin, Lluís Màrquez, & Lluís Formiga: The FAUST corpus of adequacy assessments for real-world machine translation output.  LREC 2012: Eighth international conference on Language Resources and Evaluation, 21-27 May 2012, Istanbul, Turkey; pp.29-35. [PDF, 389KB]

(2012) FAUST: feedback analysis for user adaptive statistical translation. [Project paper at] EAMT 2012: Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, May 28-30 2012, ed. Mauro Cettolo, Marcello Federico, Lucia Specia, Andy Way; p.92. [PDF, 107KB]

(2011) Miquel Esplà-Gomis, Victor M.Sánchez-Cartagena, & Juan Antonio Pérez-Ortiz: Enlarging monolingual dictionaries for machine translation with active learning and non-expert users. [RANLP 2011] Proceedings of Recent Advances in Natural Language Processing, Hissar, Bulgaria, 12-14 September 2011; pp.339-346. [PDF, 148KB]

(2011) Luciana M.Scocco: Machine translation aid to the test: a study on the benefits of post-editing for translation into the weaker language. Dissertation submitted as part requirement for Master of Arts Translation and Interpreting Studies, University of Western Sydney, 4th November 2011; 14pp. [PDF, 170KB]

(2011) Lucas Nunes Vieira & Lucia Specia: A review of translation tools from a post-editing perspective. Proceedings of the Third Joint EM+/CNGL Workshop “Bringing MT to the User: Research Meets Translators” (JEC ’11), Luxembourg, 14 October 2011; pp.33-42. [PDF, 801KB]

(2010) Yifan He, Yanjun Ma, Johann Roturier, Andy Way, & Josef van Genabith: Improving the post-editing experience using translation recommendation: a user study. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; 10pp. [PDF, 136KB]

(2010) Osamuyimen Stewart, David Lubensky, Scott Macdonald, & Julie Marcotte: Using machine translation for localization of electronic support content: evaluating end-user satisfaction. AMTA 2010: the Ninth conference of the Association for Machine Translation in the Americas, Denver, Colorado, October 31 – November 4, 2010; 6pp. [PDF, 39KB]