Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems

Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems Anya Belz editor Maja Popović editor Ehud Reiter editor Craig Thomson editor João Sedoc editor 2023-09 INCOMA Ltd., Shoumen, Bulgaria Varna, Bulgaria text conference publication humeval-2023-human https://aclanthology.org/2023.humeval-1.0/ A Manual Evaluation Method of Neural MT for Indigenous Languages Linda Wiechetek author Flammie A Pirinen author Per E Kummervold author 2023-09 text Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems Anya Belz editor Maja Popović editor Ehud Reiter editor Craig Thomson editor João Sedoc editor INCOMA Ltd., Shoumen, Bulgaria Varna, Bulgaria conference publication wiechetek-etal-2023-manual https://aclanthology.org/2023.humeval-1.1/ 2023-09 1 10 Hierarchical Evaluation Framework: Best Practices for Human Evaluation Iva Bojic author Jessica Chen author Si Yuan Chang author Qi Chwen Ong author Shafiq Joty author Josip Car author 2023-09 text Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems Anya Belz editor Maja Popović editor Ehud Reiter editor Craig Thomson editor João Sedoc editor INCOMA Ltd., Shoumen, Bulgaria Varna, Bulgaria conference publication bojic-etal-2023-hierarchical https://aclanthology.org/2023.humeval-1.2/ 2023-09 11 22 Designing a Metalanguage of Differences Between Translations: A Case Study for English-to-Japanese Translation Tomono Honda author Atsushi Fujita author Mayuka Yamamoto author Kyo Kageura author 2023-09 text Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems Anya Belz editor Maja Popović editor Ehud Reiter editor Craig Thomson editor João Sedoc editor INCOMA Ltd., Shoumen, Bulgaria Varna, Bulgaria conference publication honda-etal-2023-designing https://aclanthology.org/2023.humeval-1.3/ 2023-09 23 34 The 2023 ReproNLP Shared Task on Reproducibility of Evaluations in NLP: Overview and Results Anya Belz author Craig Thomson author 2023-09 text Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems Anya Belz editor Maja Popović editor Ehud Reiter editor Craig Thomson editor João Sedoc editor INCOMA Ltd., Shoumen, Bulgaria Varna, Bulgaria conference publication belz-thomson-2023-2023 https://aclanthology.org/2023.humeval-1.4/ 2023-09 35 48 Some lessons learned reproducing human evaluation of a data-to-text system Javier González Corbelle author Jose Alonso author Alberto Bugarín-Diz author 2023-09 text Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems Anya Belz editor Maja Popović editor Ehud Reiter editor Craig Thomson editor João Sedoc editor INCOMA Ltd., Shoumen, Bulgaria Varna, Bulgaria conference publication gonzalez-corbelle-etal-2023-lessons https://aclanthology.org/2023.humeval-1.5/ 2023-09 49 68 Unveiling NLG Human-Evaluation Reproducibility: Lessons Learned and Key Insights from Participating in the ReproNLP Challenge Lewis Watson author Dimitra Gkatzia author 2023-09 text Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems Anya Belz editor Maja Popović editor Ehud Reiter editor Craig Thomson editor João Sedoc editor INCOMA Ltd., Shoumen, Bulgaria Varna, Bulgaria conference publication watson-gkatzia-2023-unveiling https://aclanthology.org/2023.humeval-1.6/ 2023-09 69 74 How reproducible is best-worst scaling for human evaluation? A reproduction of ‘Data-to-text Generation with Macro Planning’ Emiel van Miltenburg author Anouck Braggaar author Nadine Braun author Debby Damen author Martijn Goudbeek author Chris van der Lee author Frédéric Tomas author Emiel Krahmer author 2023-09 text Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems Anya Belz editor Maja Popović editor Ehud Reiter editor Craig Thomson editor João Sedoc editor INCOMA Ltd., Shoumen, Bulgaria Varna, Bulgaria conference publication van-miltenburg-etal-2023-reproducible https://aclanthology.org/2023.humeval-1.7/ 2023-09 75 88 Human Evaluation Reproduction Report for Data-to-text Generation with Macro Planning Mohammad Arvan author Natalie Parde author 2023-09 text Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems Anya Belz editor Maja Popović editor Ehud Reiter editor Craig Thomson editor João Sedoc editor INCOMA Ltd., Shoumen, Bulgaria Varna, Bulgaria conference publication arvan-parde-2023-human https://aclanthology.org/2023.humeval-1.8/ 2023-09 89 96 Challenges in Reproducing Human Evaluation Results for Role-Oriented Dialogue Summarization Takumi Ito author Qixiang Fang author Pablo Mosteiro author Albert Gatt author Kees van Deemter author 2023-09 text Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems Anya Belz editor Maja Popović editor Ehud Reiter editor Craig Thomson editor João Sedoc editor INCOMA Ltd., Shoumen, Bulgaria Varna, Bulgaria conference publication ito-etal-2023-challenges https://aclanthology.org/2023.humeval-1.9/ 2023-09 97 123 A Reproduction Study of the Human Evaluation of Role-Oriented Dialogue Summarization Models Mingqi Gao author Jie Ruan author Xiaojun Wan author 2023-09 text Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems Anya Belz editor Maja Popović editor Ehud Reiter editor Craig Thomson editor João Sedoc editor INCOMA Ltd., Shoumen, Bulgaria Varna, Bulgaria conference publication gao-etal-2023-reproduction https://aclanthology.org/2023.humeval-1.10/ 2023-09 124 129 h_da@ReproHumn – Reproduction of Human Evaluation and Technical Pipeline Margot Mieskes author Jacob Georg Benz author 2023-09 text Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems Anya Belz editor Maja Popović editor Ehud Reiter editor Craig Thomson editor João Sedoc editor INCOMA Ltd., Shoumen, Bulgaria Varna, Bulgaria conference publication mieskes-benz-2023-h https://aclanthology.org/2023.humeval-1.11/ 2023-09 130 135 Reproducing a Comparative Evaluation of German Text-to-Speech Systems Manuela Hürlimann author Mark Cieliebak author 2023-09 text Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems Anya Belz editor Maja Popović editor Ehud Reiter editor Craig Thomson editor João Sedoc editor INCOMA Ltd., Shoumen, Bulgaria Varna, Bulgaria conference publication hurlimann-cieliebak-2023-reproducing https://aclanthology.org/2023.humeval-1.12/ 2023-09 136 144 With a Little Help from the Authors: Reproducing Human Evaluation of an MT Error Detector Ondrej Platek author Mateusz Lango author Ondrej Dusek author 2023-09 text Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems Anya Belz editor Maja Popović editor Ehud Reiter editor Craig Thomson editor João Sedoc editor INCOMA Ltd., Shoumen, Bulgaria Varna, Bulgaria conference publication platek-etal-2023-little https://aclanthology.org/2023.humeval-1.13/ 2023-09 145 152 HumEval’23 Reproduction Report for Paper 0040: Human Evaluation of Automatically Detected Over- and Undertranslations Filip Klubička author John D Kelleher author 2023-09 text Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems Anya Belz editor Maja Popović editor Ehud Reiter editor Craig Thomson editor João Sedoc editor INCOMA Ltd., Shoumen, Bulgaria Varna, Bulgaria conference publication klubicka-kelleher-2023-humeval23 https://aclanthology.org/2023.humeval-1.14/ 2023-09 153 189 Same Trends, Different Answers: Insights from a Replication Study of Human Plausibility Judgments on Narrative Continuations Yiru Li author Huiyuan Lai author Antonio Toral author Malvina Nissim author 2023-09 text Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems Anya Belz editor Maja Popović editor Ehud Reiter editor Craig Thomson editor João Sedoc editor INCOMA Ltd., Shoumen, Bulgaria Varna, Bulgaria conference publication li-etal-2023-trends https://aclanthology.org/2023.humeval-1.15/ 2023-09 190 203 Reproduction of Human Evaluations in: “It’s not Rocket Science: Interpreting Figurative Language in Narratives” Saad Mahamood author 2023-09 text Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems Anya Belz editor Maja Popović editor Ehud Reiter editor Craig Thomson editor João Sedoc editor INCOMA Ltd., Shoumen, Bulgaria Varna, Bulgaria conference publication mahamood-2023-reproduction https://aclanthology.org/2023.humeval-1.16/ 2023-09 204 209