Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²)

Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor 2025-07 Association for Computational Linguistics Vienna, Austria and virtual meeting text conference publication 979-8-89176-261-9 gem-ws-2025-1 https://aclanthology.org/2025.gem-1.0/ Towards Comprehensive Evaluation of Open-Source Language Models: A Multi-Dimensional, User-Driven Approach Qingchen Yu author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 yu-2025-towards https://aclanthology.org/2025.gem-1.1/ 2025-07 1 7 Psycholinguistic Word Features: a New Approach for the Evaluation of LLMs Alignment with Humans Javier Conde author Miguel González Saiz author María Grandury author Pedro Reviriego author Gonzalo Martínez author Marc Brysbaert author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 conde-etal-2025-psycholinguistic https://aclanthology.org/2025.gem-1.2/ 2025-07 8 17 Spatial Representation of Large Language Models in 2D Scene Wenya Wu author Weihong Deng author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 wenyawu-deng-2025-spatial https://aclanthology.org/2025.gem-1.3/ 2025-07 18 29 The Fellowship of the LLMs: Multi-Model Workflows for Synthetic Preference Optimization Dataset Generation Samee Arif author Sualeha Farid author Abdul Hameed Azeemi author Awais Athar author Agha Ali Raza author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 arif-etal-2025-fellowship https://aclanthology.org/2025.gem-1.4/ 2025-07 30 45 Does Biomedical Training Lead to Better Medical Performance? Amin Dada author Osman Alperen Koraş author Marie Bauer author Jean-Philippe Corbeil author Amanda Butler Contreras author Constantin Marc Seibold author Kaleb E Smith author Julian Friedrich author Jens Kleesiek author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 dada-etal-2025-biomedical https://aclanthology.org/2025.gem-1.5/ 2025-07 46 59 HEDS 3.0: The Human Evaluation Data Sheet Version 3.0 Anya Belz author Craig Thomson author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 belz-thomson-2025-heds https://aclanthology.org/2025.gem-1.6/ 2025-07 60 81 ARGENT: Automatic Reference-free Evaluation for Open-Ended Text Generation without Source Inputs Xinyue Zhang author Agathe Zecevic author Sebastian Zeki author Angus Roberts author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 zhang-etal-2025-argent https://aclanthology.org/2025.gem-1.8/ 2025-07 82 98 Are LLMs (Really) Ideological? An IRT-based Analysis and Alignment Tool for Perceived Socio-Economic Bias in LLMs Jasmin Wachter author Michael Radloff author Maja Smolej author Katharina Kinder-Kurlanda author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 wachter-etal-2025-llms https://aclanthology.org/2025.gem-1.9/ 2025-07 99 120 Knockout LLM Assessment: Using Large Language Models for Evaluations through Iterative Pairwise Comparisons Isik Baran Sandan author Tu Anh Dinh author Jan Niehues author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 sandan-etal-2025-knockout https://aclanthology.org/2025.gem-1.10/ 2025-07 121 128 Free-text Rationale Generation under Readability Level Control Yi-Sheng Hsu author Nils Feldhus author Sherzod Hakimov author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 hsu-etal-2025-free https://aclanthology.org/2025.gem-1.11/ 2025-07 129 150 Selective Shot Learning for Code Explanation Paheli Bhattacharya author Rishabh Gupta author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 bhattacharya-gupta-2025-selective https://aclanthology.org/2025.gem-1.12/ 2025-07 151 160 Can LLMs Detect Intrinsic Hallucinations in Paraphrasing and Machine Translation? Evangelia Gogoulou author Shorouq Zahra author Liane Guillou author Luise Dürlich author Joakim Nivre author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 gogoulou-etal-2025-llms https://aclanthology.org/2025.gem-1.13/ 2025-07 161 177 Evaluating LLMs with Multiple Problems at once Zhengxiang Wang author Jordan Kodner author Owen Rambow author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 wang-etal-2025-evaluating https://aclanthology.org/2025.gem-1.14/ 2025-07 178 199 Learning and Evaluating Factual Clarification Question Generation Without Examples Matthew Toles author Yukun Huang author Zhou Yu author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 toles-etal-2025-learning https://aclanthology.org/2025.gem-1.15/ 2025-07 200 211 SECQUE: A Benchmark for Evaluating Real-World Financial Analysis Capabilities Noga BenYoash author Menachem Brief author Oded Ovadia author Gil Shenderovitz author Moshik Mishaeli author Rachel Lemberg author Eitam Sheetrit author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 benyoash-etal-2025-secque https://aclanthology.org/2025.gem-1.16/ 2025-07 212 230 Measure only what is measurable: towards conversation requirements for evaluating task-oriented dialogue systems Emiel Van Miltenburg author Anouck Braggaar author Emmelyn Croes author Florian Kunneman author Christine Liebrecht author Gabriella Martijn author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 van-miltenburg-etal-2025-measure https://aclanthology.org/2025.gem-1.18/ 2025-07 231 238 Can Perplexity Predict Finetuning Performance? An Investigation of Tokenization Effects on Sequential Language Models for Nepali Nishant Luitel author Nirajan Bekoju author Anand Kumar Sah author Subarna Shakya author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 luitel-etal-2025-perplexity https://aclanthology.org/2025.gem-1.21/ 2025-07 239 248 Are Bias Evaluation Methods Biased ? Lina Berrayana author Sean Rooney author Luis Garcés-Erice author Ioana Giurgiu author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 berrayana-etal-2025-bias https://aclanthology.org/2025.gem-1.22/ 2025-07 249 261 IRSum: One Model to Rule Summarization and Retrieval Sotaro Takeshita author Simone Paolo Ponzetto author Kai Eckert author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 takeshita-etal-2025-irsum https://aclanthology.org/2025.gem-1.23/ 2025-07 262 275 Modeling the One-to-Many Property in Open-Domain Dialogue with LLMs Jing Yang Lee author Kong Aik Lee author Woon-Seng Gan author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 lee-etal-2025-modeling https://aclanthology.org/2025.gem-1.24/ 2025-07 276 290 Cleanse: Uncertainty Estimation Approach Using Clustering-based Semantic Consistency in LLMs Minsuh Joo author Hyunsoo Cho author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 joo-cho-2025-cleanse https://aclanthology.org/2025.gem-1.25/ 2025-07 291 301 Metric assessment protocol in the context of answer fluctuation on MCQ tasks Ekaterina Goliakova author Xavier Renard author Marie-Jeanne Lesot author Thibault Laugel author Christophe Marsala author Marcin Detyniecki author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 goliakova-etal-2025-metric https://aclanthology.org/2025.gem-1.26/ 2025-07 302 319 (Towards) Scalable Reliable Automated Evaluation with Large Language Models Bertil Braun author Martin Forell author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 braun-forell-2025-towards https://aclanthology.org/2025.gem-1.28/ 2025-07 320 336 Clustering Zero-Shot Uncertainty Estimations to Assess LLM Response Accuracy for Yes/No Q&A Christopher T Franck author Amy Vennos author W Graham Mueller author Daniel Dakota author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 franck-etal-2025-clustering https://aclanthology.org/2025.gem-1.29/ 2025-07 337 353 Using LLM Judgements for Sanity Checking Results and Reproducibility of Human Evaluations in NLP Rudali Huidrom author Anya Belz author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 huidrom-belz-2025-using https://aclanthology.org/2025.gem-1.30/ 2025-07 354 365 CoKe: Customizable Fine-Grained Story Evaluation via Chain-of-Keyword Rationalization Brihi Joshi author Sriram Venkatapathy author Mohit Bansal author Nanyun Peng author Haw-Shiuan Chang author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 joshi-etal-2025-coke https://aclanthology.org/2025.gem-1.31/ 2025-07 366 384 HuGME: A benchmark system for evaluating Hungarian generative LLMs Noémi Ligeti-Nagy author Gabor Madarasz author Flora Foldesi author Mariann Lengyel author Matyas Osvath author Bence Sarossy author Kristof Varga author Győző Zijian Yang author Enikő Héja author Tamás Váradi author Gábor Prószéky author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 ligeti-nagy-etal-2025-hugme https://aclanthology.org/2025.gem-1.32/ 2025-07 385 403 Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges Aman Singh Thakur author Kartik Choudhary author Venkat Srinik Ramayapally author Sankaran Vaidyanathan author Dieuwke Hupkes author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 thakur-etal-2025-judging https://aclanthology.org/2025.gem-1.33/ 2025-07 404 430 Analyzing the Sensitivity of Vision Language Models in Visual Question Answering Monika Shah author Sudarshan Balaji author Somdeb Sarkhel author Sanorita Dey author Deepak Venugopal author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 shah-etal-2025-analyzing https://aclanthology.org/2025.gem-1.36/ 2025-07 431 438 Investigating the Robustness of Retrieval-Augmented Generation at the Query Level Sezen Perçin author Xin Su author Qutub Sha Syed author Phillip Howard author Aleksei Kuvshinov author Leo Schwinn author Kay-Ulrich Scholl author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 percin-etal-2025-investigating https://aclanthology.org/2025.gem-1.38/ 2025-07 439 457 ELAB: Extensive LLM Alignment Benchmark in Persian Language Zahra Pourbahman author Fatemeh Rajabi author Mohammadhossein Sadeghi author Omid Ghahroodi author Somayeh Bakhshaei author Arash Amini author Reza Kazemi author Mahdieh Soleymani Baghshah author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 pourbahman-etal-2025-elab https://aclanthology.org/2025.gem-1.40/ 2025-07 458 470 Evaluating the Quality of Benchmark Datasets for Low-Resource Languages: A Case Study on Turkish Elif Ecem Umutlu author Ayse Aysu Cengiz author Ahmet Kaan Sever author Seyma Erdem author Burak Aytan author Busra Tufan author Abdullah Topraksoy author Esra Darıcı author Cagri Toraman author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 umutlu-etal-2025-evaluating https://aclanthology.org/2025.gem-1.41/ 2025-07 471 487 Big Escape Benchmark: Evaluating Human-Like Reasoning in Language Models via Real-World Escape Room Challenges Zinan Tang author QiYao Sun author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 tang-sun-2025-big https://aclanthology.org/2025.gem-1.42/ 2025-07 488 503 Event-based evaluation of abstractive news summarization Huiling You author Samia Touileb author Lilja Øvrelid author Erik Velldal author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 you-etal-2025-event https://aclanthology.org/2025.gem-1.43/ 2025-07 504 510 Fine-Tune on the Format: First Improving Multiple-Choice Evaluation for Intermediate LLM Checkpoints Alec Bunn author Sarah Wiegreffe author Ben Bogin author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 bunn-etal-2025-fine https://aclanthology.org/2025.gem-1.46/ 2025-07 511 521 PapersPlease: A Benchmark for Evaluating Motivational Values of Large Language Models Based on ERG Theory Junho Myung author Yeon Su Park author Sunwoo Kim author Shin Yoo author Alice Oh author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 myung-etal-2025-papersplease https://aclanthology.org/2025.gem-1.47/ 2025-07 522 531 Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data? Xuan Qi author Jiahao Qiu author Xinzhe Juan author Yue Wu author Mengdi Wang author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 qi-etal-2025-shallow https://aclanthology.org/2025.gem-1.48/ 2025-07 532 548 Improving Large Language Model Confidence Estimates using Extractive Rationales for Classification Jane Arleth dela Cruz author Iris Hendrickx author Martha Larson author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 cruz-etal-2025-improving https://aclanthology.org/2025.gem-1.49/ 2025-07 549 560 ReproHum #0729-04: Human Evaluation Reproduction Report for “MemSum: Extractive Summarization of Long Documents Using Multi-Step Episodic Markov Decision Processes” Simeon Junker author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 junker-2025-reprohum https://aclanthology.org/2025.gem-1.50/ 2025-07 561 567 ReproHum #0744-02: A Reproduction of the Human Evaluation of Meaning Preservation in “Factorising Meaning and Form for Intent-Preserving Paraphrasing” Julius Steen author Katja Markert author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 steen-markert-2025-reprohum https://aclanthology.org/2025.gem-1.51/ 2025-07 568 575 ReproHum #0031-01: Reproducing the Human Evaluation of Readability from “It is AI’s Turn to Ask Humans a Question” Daniel Braun author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 braun-2025-reprohum https://aclanthology.org/2025.gem-1.52/ 2025-07 576 582 ReproHum #0033-05: Human Evaluation of Factuality from A Multidisciplinary Perspective Andra-Maria Florescu author Marius Micluța-Câmpeanu author Stefana Arina Tabusca author Liviu P Dinu author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 florescu-etal-2025-reprohum https://aclanthology.org/2025.gem-1.53/ 2025-07 583 589 ReproHum: #0744-02: Investigating the Reproducibility of Semantic Preservation Human Evaluations Mohammad Arvan author Natalie Parde author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 arvan-parde-2025-reprohum https://aclanthology.org/2025.gem-1.54/ 2025-07 590 600 ReproHum #0669-08: Reproducing Sentiment Transfer Evaluation Kristýna Onderková author Mateusz Lango author Patrícia Schmidtová author Ondrej Dusek author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 onderkova-etal-2025-reprohum https://aclanthology.org/2025.gem-1.55/ 2025-07 601 608 ReproHum #0067-01: A Reproduction of the Evaluation of Cross-Lingual Summarization Supryadi author Chuang Liu author Deyi Xiong author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 supryadi-etal-2025-reprohum https://aclanthology.org/2025.gem-1.56/ 2025-07 609 614 ReproHum #0729-04: Partial reproduction of the human evaluation of the MemSum and NeuSum summarisation systems Simon Mille author Michela Lorandi author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 mille-lorandi-2025-reprohum https://aclanthology.org/2025.gem-1.57/ 2025-07 615 621 Curse of bilinguality: Evaluating monolingual and bilingual language models on Chinese linguistic benchmarks Yuwen Zhou author Yevgen Matusevych author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 zhou-matusevych-2025-curse https://aclanthology.org/2025.gem-1.58/ 2025-07 622 630 Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework Esteban Garces Arias author Hannah Blocher author Julian Rodemann author Meimingwei Li author Christian Heumann author Matthias Aßenmacher author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 garces-arias-etal-2025-towards https://aclanthology.org/2025.gem-1.59/ 2025-07 631 654 Bridging the LLM Accessibility Divide? Performance, Fairness, and Cost of Closed versus Open LLMs for Automated Essay Scoring Kezia Oketch author John P Lalor author Yi Yang author Ahmed Abbasi author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 oketch-etal-2025-bridging https://aclanthology.org/2025.gem-1.60/ 2025-07 655 669 Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages Christopher Toukmaji author Jeffrey Flanigan author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 toukmaji-flanigan-2025-prompt https://aclanthology.org/2025.gem-1.61/ 2025-07 670 704 Winning Big with Small Models: Knowledge Distillation vs. Self-Training for Reducing Hallucination in QA Agents Ashley Lewis author Michael White author Jing Liu author Toshiaki Koike-Akino author Kieran Parsons author Ye Wang author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 lewis-2025-winning https://aclanthology.org/2025.gem-1.62/ 2025-07 705 727 Ad-hoc Concept Forming in the Game Codenames as a Means for Evaluating Large Language Models Sherzod Hakimov author Lara Pfennigschmidt author David Schlangen author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 hakimov-etal-2025-ad https://aclanthology.org/2025.gem-1.63/ 2025-07 728 740 Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics Zena Al-Khalili author Nick Howell author Dietrich Klakow author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 khalili-etal-2025-evaluating https://aclanthology.org/2025.gem-1.64/ 2025-07 741 758 From Calculation to Adjudication: Examining LLM Judges on Mathematical Reasoning Tasks Andreas Stephan author Dawei Zhu author Matthias Aßenmacher author Xiaoyu Shen author Benjamin Roth author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 stephan-etal-2025-calculation https://aclanthology.org/2025.gem-1.65/ 2025-07 759 773 PersonaTwin: A Multi-Tier Prompt Conditioning Framework for Generating and Evaluating Personalized Digital Twins Sihan Chen author John P Lalor author Yi Yang author Ahmed Abbasi author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 chen-etal-2025-personatwin https://aclanthology.org/2025.gem-1.66/ 2025-07 774 788 Coreference as an indicator of context scope in multimodal narrative Nikolai Ilinykh author Shalom Lappin author Asad B Sayeed author Sharid Loáiciga author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 ilinykh-etal-2025-coreference https://aclanthology.org/2025.gem-1.67/ 2025-07 789 807 PATCH! Psychometrics-AssisTed BenCHmarking of Large Language Models against Human Populations: A Case Study of Proficiency in 8th Grade Mathematics Qixiang Fang author Daniel Oberski author Dong Nguyen author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 fang-etal-2025-patch https://aclanthology.org/2025.gem-1.68/ 2025-07 808 823 MCQFormatBench: Robustness Tests for Multiple-Choice Questions Hiroo Takizawa author Saku Sugawara author Akiko Aizawa author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 takizawa-etal-2025-mcqformatbench https://aclanthology.org/2025.gem-1.69/ 2025-07 824 846 (Dis)improved?! How Simplified Language Affects Large Language Model Performance across Languages Miriam Anschütz author Anastasiya Damaratskaya author Chaeeun Joy Lee author Arthur Schmalz author Edoardo Mosca author Georg Groh author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 anschutz-etal-2025-dis https://aclanthology.org/2025.gem-1.70/ 2025-07 847 861 Fine-Grained Constraint Generation-Verification for Improved Instruction-Following Zhixiang Liang author Zhenyu Hou author Xiao Wang author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 liang-etal-2025-fine https://aclanthology.org/2025.gem-1.71/ 2025-07 862 879 Finance Language Model Evaluation (FLaME) Glenn Matlin author Mika Okamoto author Huzaifa Pardawala author Yang Yang author Sudheer Chava author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 matlin-etal-2025-finance https://aclanthology.org/2025.gem-1.72/ 2025-07 880 926 sPhinX: Sample Efficient Multilingual Instruction Fine-Tuning Through N-shot Guided Prompting Sanchit Ahuja author Kumar Tanmay author Hardik Hansrajbhai Chauhan author Barun Patra author Kriti Aggarwal author Luciano Del Corro author Arindam Mitra author Tejas Indulal Dhamecha author Ahmed Hassan Awadallah author Monojit Choudhury author Vishrav Chaudhary author Sunayana Sitaram author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 ahuja-etal-2025-sphinx https://aclanthology.org/2025.gem-1.73/ 2025-07 927 946 Single- vs. Dual-Prompt Dialogue Generation with LLMs for Job Interviews in Human Resources Joachim De Baer author A Seza Doğruöz author Thomas Demeester author Chris Develder author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 de-baer-etal-2025-single https://aclanthology.org/2025.gem-1.74/ 2025-07 947 957 Natural Language Counterfactual Explanations in Financial Text Classification: A Comparison of Generators and Evaluation Metrics Karol Dobiczek author Patrick Altmeyer author Cynthia C S Liem author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 dobiczek-etal-2025-natural https://aclanthology.org/2025.gem-1.75/ 2025-07 958 972 An Analysis of Datasets, Metrics and Models in Keyphrase Generation Florian Boudin author Akiko Aizawa author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 boudin-aizawa-2025-analysis https://aclanthology.org/2025.gem-1.76/ 2025-07 973 973 U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in Large Language Models Konstantin Chernyshev author Vitaliy Polshkov author Vlad Stepanov author Alex Myasnikov author Ekaterina Artemova author Alexei Miasnikov author Sergei Tilga author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 chernyshev-etal-2025-u https://aclanthology.org/2025.gem-1.77/ 2025-07 974 1001 The 2025 ReproNLP Shared Task on Reproducibility of Evaluations in NLP: Overview and Results Anya Belz author Craig Thomson author Javier González Corbelle author Malo Ruelle author 2025-07 text Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) Ofir Arviv editor Miruna Clinciu editor Kaustubh Dhole editor Rotem Dror editor Sebastian Gehrmann editor Eliya Habba editor Itay Itzhak editor Simon Mille editor Yotam Perlitz editor Enrico Santus editor João Sedoc editor Michal Shmueli Scheuer editor Gabriel Stanovsky editor Oyvind Tafjord editor Association for Computational Linguistics Vienna, Austria and virtual meeting conference publication 979-8-89176-261-9 belz-etal-2025-2025 https://aclanthology.org/2025.gem-1.78/ 2025-07 1002 1016