Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems

Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor 2021-11 Association for Computational Linguistics Punta Cana, Dominican Republic text conference publication eval4nlp-2021-evaluation https://aclanthology.org/2021.eval4nlp-1.0/ Differential Evaluation: a Qualitative Analysis of Natural Language Processing System Behavior Based Upon Data Resistance to Processing Lucie Gianola author Hicham El Boukkouri author Cyril Grouin author Thomas Lavergne author Patrick Paroubek author Pierre Zweigenbaum author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication gianola-etal-2021-differential 10.18653/v1/2021.eval4nlp-1.1 https://aclanthology.org/2021.eval4nlp-1.1/ 2021-11 1 10 Validating Label Consistency in NER Data Annotation Qingkai Zeng author Mengxia Yu author Wenhao Yu author Tianwen Jiang author Meng Jiang author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication zeng-etal-2021-validating 10.18653/v1/2021.eval4nlp-1.2 https://aclanthology.org/2021.eval4nlp-1.2/ 2021-11 11 15 How Emotionally Stable is ALBERT? Testing Robustness with Stochastic Weight Averaging on a Sentiment Analysis Task Urja Khurana author Eric Nalisnick author Antske Fokkens author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication khurana-etal-2021-emotionally 10.18653/v1/2021.eval4nlp-1.3 https://aclanthology.org/2021.eval4nlp-1.3/ 2021-11 16 31 StoryDB: Broad Multi-language Narrative Dataset Alexey Tikhonov author Igor Samenko author Ivan P Yamshchikov author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication tikhonov-etal-2021-storydb 10.18653/v1/2021.eval4nlp-1.4 https://aclanthology.org/2021.eval4nlp-1.4/ 2021-11 32 39 SeqScore: Addressing Barriers to Reproducible Named Entity Recognition Evaluation Chester Palen-Michel author Nolan Holley author Constantine Lignos author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication palen-michel-etal-2021-seqscore 10.18653/v1/2021.eval4nlp-1.5 https://aclanthology.org/2021.eval4nlp-1.5/ 2021-11 40 50 Trainable Ranking Models to Evaluate the Semantic Accuracy of Data-to-Text Neural Generator Nicolas Garneau author Luc Lamontagne author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication garneau-lamontagne-2021-trainable 10.18653/v1/2021.eval4nlp-1.6 https://aclanthology.org/2021.eval4nlp-1.6/ 2021-11 51 61 Evaluation of Unsupervised Automatic Readability Assessors Using Rank Correlations Yo Ehara author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication ehara-2021-evaluation 10.18653/v1/2021.eval4nlp-1.7 https://aclanthology.org/2021.eval4nlp-1.7/ 2021-11 62 72 Testing Cross-Database Semantic Parsers With Canonical Utterances Heather Lent author Semih Yavuz author Tao Yu author Tong Niu author Yingbo Zhou author Dragomir Radev author Xi Victoria Lin author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication lent-etal-2021-testing 10.18653/v1/2021.eval4nlp-1.8 https://aclanthology.org/2021.eval4nlp-1.8/ 2021-11 73 83 Writing Style Author Embedding Evaluation Enzo Terreau author Antoine Gourru author Julien Velcin author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication terreau-etal-2021-writing 10.18653/v1/2021.eval4nlp-1.9 https://aclanthology.org/2021.eval4nlp-1.9/ 2021-11 84 93 ESTIME: Estimation of Summary-to-Text Inconsistency by Mismatched Embeddings Oleg Vasilyev author John Bohannon author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication vasilyev-bohannon-2021-estime 10.18653/v1/2021.eval4nlp-1.10 https://aclanthology.org/2021.eval4nlp-1.10/ 2021-11 94 103 Statistically Significant Detection of Semantic Shifts using Contextual Word Embeddings Yang Liu author Alan Medlar author Dorota Glowacka author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication liu-etal-2021-statistically 10.18653/v1/2021.eval4nlp-1.11 https://aclanthology.org/2021.eval4nlp-1.11/ 2021-11 104 113 Referenceless Parsing-Based Evaluation of AMR-to-English Generation Emma Manning author Nathan Schneider author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication manning-schneider-2021-referenceless 10.18653/v1/2021.eval4nlp-1.12 https://aclanthology.org/2021.eval4nlp-1.12/ 2021-11 114 122 MIPE: A Metric Independent Pipeline for Effective Code-Mixed NLG Evaluation Ayush Garg author Sammed Kagi author Vivek Srivastava author Mayank Singh author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication garg-etal-2021-mipe 10.18653/v1/2021.eval4nlp-1.13 https://aclanthology.org/2021.eval4nlp-1.13/ 2021-11 123 132 IST-Unbabel 2021 Submission for the Explainable Quality Estimation Shared Task Marcos Treviso author Nuno M Guerreiro author Ricardo Rei author André F T Martins author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication treviso-etal-2021-ist 10.18653/v1/2021.eval4nlp-1.14 https://aclanthology.org/2021.eval4nlp-1.14/ 2021-11 133 145 Error Identification for Machine Translation with Metric Embedding and Attention Raphael Rubino author Atsushi Fujita author Benjamin Marie author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication rubino-etal-2021-error 10.18653/v1/2021.eval4nlp-1.15 https://aclanthology.org/2021.eval4nlp-1.15/ 2021-11 146 156 Reference-Free Word- and Sentence-Level Translation Evaluation with Token-Matching Metrics Christoph Wolfgang Leiter author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication leiter-2021-reference 10.18653/v1/2021.eval4nlp-1.16 https://aclanthology.org/2021.eval4nlp-1.16/ 2021-11 157 164 The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Results Marina Fomicheva author Piyawat Lertvittayakumjorn author Wei Zhao author Steffen Eger author Yang Gao author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication fomicheva-etal-2021-eval4nlp 10.18653/v1/2021.eval4nlp-1.17 https://aclanthology.org/2021.eval4nlp-1.17/ 2021-11 165 178 Developing a Benchmark for Reducing Data Bias in Authorship Attribution Benjamin Murauer author Günther Specht author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication murauer-specht-2021-developing 10.18653/v1/2021.eval4nlp-1.18 https://aclanthology.org/2021.eval4nlp-1.18/ 2021-11 179 188 Error-Sensitive Evaluation for Ordinal Target Variables David Chen author Maury Courtland author Adam Faulkner author Aysu Ezen-Can author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication chen-etal-2021-error 10.18653/v1/2021.eval4nlp-1.19 https://aclanthology.org/2021.eval4nlp-1.19/ 2021-11 189 199 HinGE: A Dataset for Generation and Evaluation of Code-Mixed Hinglish Text Vivek Srivastava author Mayank Singh author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication srivastava-singh-2021-hinge 10.18653/v1/2021.eval4nlp-1.20 https://aclanthology.org/2021.eval4nlp-1.20/ 2021-11 200 208 What is SemEval evaluating? A Systematic Analysis of Evaluation Campaigns in NLP Oskar Wysocki author Malina Florea author Dónal Landers author André Freitas author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication wysocki-etal-2021-semeval 10.18653/v1/2021.eval4nlp-1.21 https://aclanthology.org/2021.eval4nlp-1.21/ 2021-11 209 229 The UMD Submission to the Explainable MT Quality Estimation Shared Task: Combining Explanation Models with Sequence Labeling Tasnim Kabir author Marine Carpuat author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication kabir-carpuat-2021-umd 10.18653/v1/2021.eval4nlp-1.22 https://aclanthology.org/2021.eval4nlp-1.22/ 2021-11 230 237 Explaining Errors in Machine Translation with Absolute Gradient Ensembles Melda Eksi author Erik Gelbing author Jonathan Stieber author Chi Viet Vu author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication eksi-etal-2021-explaining 10.18653/v1/2021.eval4nlp-1.23 https://aclanthology.org/2021.eval4nlp-1.23/ 2021-11 238 249 Explainable Quality Estimation: CUNI Eval4NLP Submission Peter Polák author Muskaan Singh author Ondřej Bojar author 2021-11 text Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems Yang Gao editor Steffen Eger editor Wei Zhao editor Piyawat Lertvittayakumjorn editor Marina Fomicheva editor Association for Computational Linguistics Punta Cana, Dominican Republic conference publication polak-etal-2021-explainable 10.18653/v1/2021.eval4nlp-1.24 https://aclanthology.org/2021.eval4nlp-1.24/ 2021-11 250 255