EnCBP: A New Benchmark Dataset for Finer-Grained Cultural Background Prediction in English

Weicheng Ma, Samiha Datta, Lili Wang, Soroush Vosoughi


Abstract
While cultural backgrounds have been shown to affect linguistic expressions, existing natural language processing (NLP) research on culture modeling is overly coarse-grained and does not examine cultural differences among speakers of the same language. To address this problem and augment NLP models with cultural background features, we collect, annotate, manually validate, and benchmark EnCBP, a finer-grained news-based cultural background prediction dataset in English. Through language modeling (LM) evaluations and manual analyses, we confirm that there are noticeable differences in linguistic expressions among five English-speaking countries and across four states in the US. Additionally, our evaluations on nine syntactic (CoNLL-2003), semantic (PAWS-Wiki, QNLI, STS-B, and RTE), and psycholinguistic tasks (SST-5, SST-2, Emotion, and Go-Emotions) show that, while introducing cultural background information does not benefit the Go-Emotions task due to text domain conflicts, it noticeably improves deep learning (DL) model performance on other tasks. Our findings strongly support the importance of cultural background modeling to a wide variety of NLP tasks and demonstrate the applicability of EnCBP in culture-related research.
Anthology ID:
2022.findings-acl.221
Volume:
Findings of the Association for Computational Linguistics: ACL 2022
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2811–2823
Language:
URL:
https://aclanthology.org/2022.findings-acl.221
DOI:
10.18653/v1/2022.findings-acl.221
Bibkey:
Cite (ACL):
Weicheng Ma, Samiha Datta, Lili Wang, and Soroush Vosoughi. 2022. EnCBP: A New Benchmark Dataset for Finer-Grained Cultural Background Prediction in English. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2811–2823, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
EnCBP: A New Benchmark Dataset for Finer-Grained Cultural Background Prediction in English (Ma et al., Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-acl.221.pdf
Data
CARERCoNLL 2003GLUEGoEmotionsQNLISSTSST-2SST-5