Evaluating Contextual Embeddings and their Extraction Layers for Depression Assessment

Matthew Matero, Albert Hung, H. Andrew Schwartz


Abstract
Many recent works in natural language processing have demonstrated ability to assess aspects of mental health from personal discourse. At the same time, pre-trained contextual word embedding models have grown to dominate much of NLP but little is known empirically on how to best apply them for mental health assessment. Using degree of depression as a case study, we do an empirical analysis on which off-the-shelf language model, individual layers, and combinations of layers seem most promising when applied to human-level NLP tasks. Notably, we find RoBERTa most effective and, despite the standard in past work suggesting the second-to-last or concatenation of the last 4 layers, we find layer 19 (sixth-to last) is at least as good as layer 23 when using 1 layer. Further, when using multiple layers, distributing them across the second half (i.e. Layers 12+), rather than last 4, of the 24 layers yielded the most accurate results.
Anthology ID:
2022.wassa-1.9
Volume:
Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Jeremy Barnes, Orphée De Clercq, Valentin Barriere, Shabnam Tafreshi, Sawsan Alqahtani, João Sedoc, Roman Klinger, Alexandra Balahur
Venue:
WASSA
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
89–94
Language:
URL:
https://aclanthology.org/2022.wassa-1.9
DOI:
10.18653/v1/2022.wassa-1.9
Bibkey:
Cite (ACL):
Matthew Matero, Albert Hung, and H. Andrew Schwartz. 2022. Evaluating Contextual Embeddings and their Extraction Layers for Depression Assessment. In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, pages 89–94, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Evaluating Contextual Embeddings and their Extraction Layers for Depression Assessment (Matero et al., WASSA 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.wassa-1.9.pdf