Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework

Anna Wegmann, Dong Nguyen


Abstract
Style is an integral part of natural language. However, evaluation methods for style measures are rare, often task-specific and usually do not control for content. We propose the modular, fine-grained and content-controlled similarity-based STyle EvaLuation framework (STEL) to test the performance of any model that can compare two sentences on style. We illustrate STEL with two general dimensions of style (formal/informal and simple/complex) as well as two specific characteristics of style (contrac’tion and numb3r substitution). We find that BERT-based methods outperform simple versions of commonly used style measures like 3-grams, punctuation frequency and LIWC-based approaches. We invite the addition of further tasks and task instances to STEL and hope to facilitate the improvement of style-sensitive measures.
Anthology ID:
2021.emnlp-main.569
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7109–7130
Language:
URL:
https://aclanthology.org/2021.emnlp-main.569
DOI:
10.18653/v1/2021.emnlp-main.569
Bibkey:
Cite (ACL):
Anna Wegmann and Dong Nguyen. 2021. Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7109–7130, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework (Wegmann & Nguyen, EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.569.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.569.mp4
Code
 nlpsoc/stel