ClusterDataSplit: Exploring Challenging Clustering-Based Data Splits for Model Performance Evaluation

Hanna Wecker, Annemarie Friedrich, Heike Adel


Abstract
This paper adds to the ongoing discussion in the natural language processing community on how to choose a good development set. Motivated by the real-life necessity of applying machine learning models to different data distributions, we propose a clustering-based data splitting algorithm. It creates development (or test) sets which are lexically different from the training data while ensuring similar label distributions. Hence, we are able to create challenging cross-validation evaluation setups while abstracting away from performance differences resulting from label distribution shifts between training and test data. In addition, we present a Python-based tool for analyzing and visualizing data split characteristics and model performance. We illustrate the workings and results of our approach using a sentiment analysis and a patent classification task.
Anthology ID:
2020.eval4nlp-1.15
Volume:
Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems
Month:
November
Year:
2020
Address:
Online
Venues:
EMNLP | Eval4NLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
155–163
Language:
URL:
https://aclanthology.org/2020.eval4nlp-1.15
DOI:
10.18653/v1/2020.eval4nlp-1.15
Bibkey:
Cite (ACL):
Hanna Wecker, Annemarie Friedrich, and Heike Adel. 2020. ClusterDataSplit: Exploring Challenging Clustering-Based Data Splits for Model Performance Evaluation. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 155–163, Online. Association for Computational Linguistics.
Cite (Informal):
ClusterDataSplit: Exploring Challenging Clustering-Based Data Splits for Model Performance Evaluation (Wecker et al., Eval4NLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.eval4nlp-1.15.pdf
Video:
 https://slideslive.com/38939708
Code
 boschresearch/clusterdatasplit_eval4nlp-2020
Data
SST