%0 Conference Proceedings %T Does It Capture STEL? A Modular, Similarity-based Linguistic Style Evaluation Framework %A Wegmann, Anna %A Nguyen, Dong %Y Moens, Marie-Francine %Y Huang, Xuanjing %Y Specia, Lucia %Y Yih, Scott Wen-tau %S Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing %D 2021 %8 November %I Association for Computational Linguistics %C Online and Punta Cana, Dominican Republic %F wegmann-nguyen-2021-capture %X Style is an integral part of natural language. However, evaluation methods for style measures are rare, often task-specific and usually do not control for content. We propose the modular, fine-grained and content-controlled similarity-based STyle EvaLuation framework (STEL) to test the performance of any model that can compare two sentences on style. We illustrate STEL with two general dimensions of style (formal/informal and simple/complex) as well as two specific characteristics of style (contrac’tion and numb3r substitution). We find that BERT-based methods outperform simple versions of commonly used style measures like 3-grams, punctuation frequency and LIWC-based approaches. We invite the addition of further tasks and task instances to STEL and hope to facilitate the improvement of style-sensitive measures. %R 10.18653/v1/2021.emnlp-main.569 %U https://aclanthology.org/2021.emnlp-main.569 %U https://doi.org/10.18653/v1/2021.emnlp-main.569 %P 7109-7130