Maurizio Atzori
2024
Assessing Italian Large Language Models on Energy Feedback Generation: A Human Evaluation Study
Manuela Sanguinetti
|
Alessandro Pani
|
Alessandra Perniciano
|
Luca Zedda
|
Andrea Loddo
|
Maurizio Atzori
Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024)
This work presents a comparison of some recently-released instruction-tuned large language models for Italian, focusing in particular on their effectiveness in a specific application scenario, i.e., that of delivering energy feedback. This work is part of a larger project aimed at developing a conversational interface for users of a renewable energy community, where clarity and accuracy of the provided feedback are important for a proper energy management. This comparison is based on the human evaluation of the output produced by such models using energy data as input. Specifically, the data pertains to information regarding the power flows within a household equipped with a photovoltaic (PV) plant and a battery storage system. The goal of the feedback is precisely that of providing the user with such information in a meaningful way based on the specific aspect they intend to monitor at a given moment (e.g., self-consumption levels, the power generated by the PV panels or imported from the main grid, or the battery state of charge). This evaluation experiment has the two-fold purpose of providing an exploratory analysis of the models’ abilities on this specific generation task solely relying on the information and instruction provided in the prompt, and as an initial investigation into their potential as reliable tools for generating user-friendly energy feedback in this intended scenario.
Snarci at SemEval-2024 Task 4: Themis Model for Binary Classification of Memes
Luca Zedda
|
Alessandra Perniciano
|
Andrea Loddo
|
Cecilia Di Ruberto
|
Manuela Sanguinetti
|
Maurizio Atzori
Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)
This paper introduces an approach developed for multimodal meme analysis, specifically targeting the identification of persuasion techniques embedded within memes. Our methodology integrates Large Language Models (LLMs) and contrastive learning image encoders to discern the presence of persuasive elements in memes across diverse platforms. By capitalizing on the contextual understanding facilitated by LLMs and the discriminative power of contrastive learning for image encoding, our framework provides a robust solution for detecting and classifying memes with persuasion techniques. The system was used in Task 4 of Semeval 2024, precisely for Substask 2b (binary classification of presence of persuasion techniques). It showed promising results overall, achieving a Macro-F1=0.7986 on the English test data (i.e., the language the system was trained on) and Macro-F1=0.66777/0.47917/0.5554, respectively, on the other three “surprise” languages proposed by the task organizers, i.e., Bulgarian, North Macedonian and Arabic. The paper provides an overview of the system, along with a discussion of the results obtained and its main limitations.
Search
Fix data
Co-authors
- Andrea Loddo 2
- Alessandra Perniciano 2
- Manuela Sanguinetti 2
- Luca Zedda 2
- Cecilia Di Ruberto 1
- show all...