Lon-eå at SemEval-2023 Task 11: A Comparison of Activation Functions for Soft and Hard Label Prediction

Peyman Hosseini, Mehran Hosseini, Sana Al-azzawi, Marcus Liwicki, Ignacio Castro, Matthew Purver


Abstract
We study the influence of different activation functions in the output layer of pre-trained transformer models for soft and hard label prediction in the learning with disagreement task. In this task, the goal is to quantify the amount of disagreement via predicting soft labels. To predict the soft labels, we use BERT-based preprocessors and encoders and vary the activation function used in the output layer, while keeping other parameters constant. The soft labels are then used for the hard label prediction. The activation functions considered are sigmoid as well as a step-function that is added to the model post-training and a sinusoidal activation function, which is introduced for the first time in this paper.
Anthology ID:
2023.semeval-1.185
Volume:
Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)
Month:
July
Year:
2023
Address:
Toronto, Canada
Editors:
Atul Kr. Ojha, A. Seza Doğruöz, Giovanni Da San Martino, Harish Tayyar Madabushi, Ritesh Kumar, Elisa Sartori
Venue:
SemEval
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
1329–1334
Language:
URL:
https://aclanthology.org/2023.semeval-1.185
DOI:
10.18653/v1/2023.semeval-1.185
Bibkey:
Cite (ACL):
Peyman Hosseini, Mehran Hosseini, Sana Al-azzawi, Marcus Liwicki, Ignacio Castro, and Matthew Purver. 2023. Lon-eå at SemEval-2023 Task 11: A Comparison of Activation Functions for Soft and Hard Label Prediction. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1329–1334, Toronto, Canada. Association for Computational Linguistics.
Cite (Informal):
Lon-eå at SemEval-2023 Task 11: A Comparison of Activation Functions for Soft and Hard Label Prediction (Hosseini et al., SemEval 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.semeval-1.185.pdf