Environmental Impact Measurement in the MentalRiskES Evaluation Campaign

Alba M. Mármol Romero, Adrián Moreno-Muñoz, Flor Miriam Plaza-del-Arco, M. Dolores Molina González, Arturo Montejo-Ráez


Abstract
With the rise of Large Language Models (LLMs), the NLP community is increasingly aware of the environmental consequences of model development due to the energy consumed for training and running these models. This study investigates the energy consumption and environmental impact of systems participating in the MentalRiskES shared task, at the Iberian Language Evaluation Forum (IberLEF) in the year 2023, which focuses on early risk identification of mental disorders in Spanish comments. Participants were asked to submit, for each prediction, a set of efficiency metrics, being carbon dioxide emissions among them. We conduct an empirical analysis of the data submitted considering model architecture, task complexity, and dataset characteristics, covering a spectrum from traditional Machine Learning (ML) models to advanced LLMs. Our findings contribute to understanding the ecological footprint of NLP systems and advocate for prioritizing environmental impact assessment in shared tasks to foster sustainability across diverse model types and approaches, being evaluation campaigns an adequate framework for this kind of analysis.
Anthology ID:
2024.tdle-1.6
Volume:
Proceedings of the Second International Workshop Towards Digital Language Equality (TDLE): Focusing on Sustainability @ LREC-COLING 2024
Month:
May
Year:
2024
Address:
Torino, Italia
Editors:
Federico Gaspari, Joss Moorkens, Itziar Aldabe, Aritz Farwell, Begona Altuna, Stelios Piperidis, Georg Rehm, German Rigau
Venues:
TDLE | WS
SIG:
Publisher:
ELRA and ICCL
Note:
Pages:
61–72
Language:
URL:
https://aclanthology.org/2024.tdle-1.6
DOI:
Bibkey:
Cite (ACL):
Alba M. Mármol Romero, Adrián Moreno-Muñoz, Flor Miriam Plaza-del-Arco, M. Dolores Molina González, and Arturo Montejo-Ráez. 2024. Environmental Impact Measurement in the MentalRiskES Evaluation Campaign. In Proceedings of the Second International Workshop Towards Digital Language Equality (TDLE): Focusing on Sustainability @ LREC-COLING 2024, pages 61–72, Torino, Italia. ELRA and ICCL.
Cite (Informal):
Environmental Impact Measurement in the MentalRiskES Evaluation Campaign (Mármol Romero et al., TDLE-WS 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.tdle-1.6.pdf