Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)

Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor 2023-12 Association for Computational Linguistics Singapore text conference publication gem-2023-natural https://aclanthology.org/2023.gem-1.0/ Contextualizing the Limits of Model & Evaluation Dataset Curation on Semantic Similarity Classification Tasks Daniel Theron author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication theron-2023-contextualizing https://aclanthology.org/2023.gem-1.1/ 2023-12 1 8 Dialogue Quality and Emotion Annotations for Customer Support Conversations John Mendonca author Patrícia Pereira author Miguel Menezes author Vera Cabarrão author Ana C Farinha author Helena Moniz author Alon Lavie author Isabel Trancoso author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication mendonca-etal-2023-dialogue https://aclanthology.org/2023.gem-1.2/ 2023-12 9 21 Formalizing content creation and evaluation methods for AI-generated social media content Christian Jensen author Axel Højmark author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication jensen-hojmark-2023-formalizing https://aclanthology.org/2023.gem-1.3/ 2023-12 22 41 Automatic Evaluation of Generative Models with Instruction Tuning Shuhaib Mehri author Vered Shwartz author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication mehri-shwartz-2023-automatic https://aclanthology.org/2023.gem-1.4/ 2023-12 42 52 Effective Proxy for Human Labeling: Ensemble Disagreement Scores in Large Language Models for Industrial NLP Wei Du author Laksh Advani author Yashmeet Gambhir author Daniel Perry author Prashant Shiralkar author Zhengzheng Xing author Aaron Colak author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication du-etal-2023-effective https://aclanthology.org/2023.gem-1.5/ 2023-12 53 61 Automatic Reflection Generation for Peer-to-Peer Counseling Emma O’neil author João Sedoc author Diyi Yang author Haiyi Zhu author Lyle Ungar author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication oneil-etal-2023-automatic https://aclanthology.org/2023.gem-1.6/ 2023-12 62 75 One-Shot and Few-Shot Exemplification Modeling John Harvill author Hee Suk Yoon author Eunseop Yoon author Mark Hasegawa-Johnson author Chang Yoo author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication harvill-etal-2023-one-shot https://aclanthology.org/2023.gem-1.7/ 2023-12 76 87 Leveraging Large Language Models for Enhanced Product Descriptions in eCommerce Jianghong Zhou author Bo Liu author Jhalak Acharya author Yao Hong author Kuang-Chih Lee author Musen Wen author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication zhou-etal-2023-leveraging https://aclanthology.org/2023.gem-1.8/ 2023-12 88 96 QAMPARI: A Benchmark for Open-domain Questions with Many Answers Samuel Amouyal author Tomer Wolfson author Ohad Rubin author Ori Yoran author Jonathan Herzig author Jonathan Berant author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication amouyal-etal-2023-qampari https://aclanthology.org/2023.gem-1.9/ 2023-12 97 110 Unveiling Safety Vulnerabilities of Large Language Models George Kour author Marcel Zalmanovici author Naama Zwerdling author Esther Goldbraich author Ora Fandina author Ateret Anaby Tavor author Orna Raz author Eitan Farchi author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication kour-etal-2023-unveiling https://aclanthology.org/2023.gem-1.10/ 2023-12 111 127 Adapting Pre-trained Generative Models for Extractive Question Answering Prabir Mallick author Tapas Nayak author Indrajit Bhattacharya author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication mallick-etal-2023-adapting https://aclanthology.org/2023.gem-1.11/ 2023-12 128 137 Predicting Question-Answering Performance of Large Language Models through Semantic Consistency Ella Rabinovich author Samuel Ackerman author Orna Raz author Eitan Farchi author Ateret Anaby Tavor author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication rabinovich-etal-2023-predicting https://aclanthology.org/2023.gem-1.12/ 2023-12 138 154 Towards Effective Long-Form QA with Evidence Augmentation Mengxia Yu author Sara Rosenthal author Mihaela Bornea author Avi Sil author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication yu-etal-2023-towards https://aclanthology.org/2023.gem-1.13/ 2023-12 155 164 Harnessing the Plug-and-Play Controller by Prompting Hao Wang author Lei Sha author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication wang-sha-2023-harnessing https://aclanthology.org/2023.gem-1.14/ 2023-12 165 174 Context and Literacy Aware Learnable Metric for Text Simplification Jeongwon Kwak author Hyeryun Park author Kyungmo Kim author Jinwook Choi author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication kwak-etal-2023-context https://aclanthology.org/2023.gem-1.15/ 2023-12 175 180 Synthetic Dialogue Dataset Generation using LLM Agents Yelaman Abdullin author Diego Molla author Bahadorreza Ofoghi author John Yearwood author Qingyang Li author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication abdullin-etal-2023-synthetic https://aclanthology.org/2023.gem-1.16/ 2023-12 181 191 An Empirical Bayes Framework for Open-Domain Dialogue Generation Jing Yang Lee author Kong Aik Lee author Woon Seng Gan author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication lee-etal-2023-empirical https://aclanthology.org/2023.gem-1.17/ 2023-12 192 204 Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Models Joseph Marvin Imperial author Harish Tayyar Madabushi author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication imperial-tayyar-madabushi-2023-flesch https://aclanthology.org/2023.gem-1.18/ 2023-12 205 223 ChatGPT as a Java Decompiler Bradley Mcdanel author Zhanhao Liu author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication mcdanel-liu-2023-chatgpt https://aclanthology.org/2023.gem-1.19/ 2023-12 224 232 Multi-domain Summarization from Leaderboards to Practice: Re-examining Automatic and Human Evaluation David Demeter author Oshin Agarwal author Simon Ben Igeri author Marko Sterbentz author Neil Molino author John Conroy author Ani Nenkova author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication demeter-etal-2023-multi https://aclanthology.org/2023.gem-1.20/ 2023-12 233 242 Targeted Image Data Augmentation Increases Basic Skills Captioning Robustness Valentin Barriere author Felipe Del Rio author Andres Carvallo author Carlos Aspillaga author Eugenio Herrera-Berg author Cristian Buc author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication barriere-etal-2023-targeted https://aclanthology.org/2023.gem-1.21/ 2023-12 243 257 Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses Xenia Ohmer author Elia Bruni author Dieuwke Hupkes author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication ohmer-etal-2023-separating https://aclanthology.org/2023.gem-1.22/ 2023-12 258 276 Text Encoders Lack Knowledge: Leveraging Generative LLMs for Domain-Specific Semantic Textual Similarity Joseph Gatto author Omar Sharif author Parker Seegmiller author Philip Bohlman author Sarah M Preum author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication gatto-etal-2023-text https://aclanthology.org/2023.gem-1.23/ 2023-12 277 288 To Burst or Not to Burst: Generating and Quantifying Improbable Text Kuleen Sasse author Efsun Sarioglu Kayi author Samuel Barham author Edward Staley author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication sasse-etal-2023-burst https://aclanthology.org/2023.gem-1.24/ 2023-12 289 309 Are Large Language Models Reliable Judges? A Study on the Factuality Evaluation Capabilities of LLMs Xue-Yong Fu author Md Tahmid Rahman Laskar author Cheng Chen author Shashi Bhushan Tn author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication fu-etal-2023-large https://aclanthology.org/2023.gem-1.25/ 2023-12 310 316 RankAug: Augmented data ranking for text classification Tiasa Roy author Priyam Basu author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication roy-basu-2023-rankaug https://aclanthology.org/2023.gem-1.26/ 2023-12 317 323 Separating the Wheat from the Chaff with BREAD: An open-source benchmark and metrics to detect redundancy in text Isaac Caswell author Lisa Wang author Isabel Papadimitriou author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication caswell-etal-2023-separating https://aclanthology.org/2023.gem-1.27/ 2023-12 324 338 Elo Uncovered: Robustness and Best Practices in Language Model Evaluation Meriem Boubdir author Edward Kim author Beyza Ermis author Sara Hooker author Marzieh Fadaee author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication boubdir-etal-2023-elo https://aclanthology.org/2023.gem-1.28/ 2023-12 339 352 PersonalityChat: Conversation Distillation for Personalized Dialog Modeling with Facts and Traits Ehsan Lotfi author Maxime De Bruyn author Jeska Buhmann author Walter Daelemans author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication lotfi-etal-2023-personalitychat https://aclanthology.org/2023.gem-1.29/ 2023-12 353 371 How well ChatGPT understand Malaysian English? An Evaluation on Named Entity Recognition and Relation Extraction Mohanraj Chanthran author Lay-Ki Soon author Ong Huey Fang author Bhawani Selvaretnam author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication chanthran-etal-2023-well https://aclanthology.org/2023.gem-1.30/ 2023-12 372 397 Post Turing: Mapping the landscape of LLM Evaluation Alexey Tikhonov author Ivan P Yamshchikov author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication tikhonov-yamshchikov-2023-post https://aclanthology.org/2023.gem-1.31/ 2023-12 398 412 A Simple yet Efficient Ensemble Approach for AI-generated Text Detection Harika Abburi author Kalyani Roy author Michael Suesserman author Nirmala Pudota author Balaji Veeramani author Edward Bowen author Sanmitra Bhattacharya author 2023-12 text Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM) Sebastian Gehrmann editor Alex Wang editor João Sedoc editor Elizabeth Clark editor Kaustubh Dhole editor Khyathi Raghavi Chandu editor Enrico Santus editor Hooman Sedghamiz editor Association for Computational Linguistics Singapore conference publication abburi-etal-2023-simple https://aclanthology.org/2023.gem-1.32/ 2023-12 413 421