A Perspectivist Corpus of Numbers in Social Judgements

Marlon May, Lucie Flek, Charles Welch


Abstract
With growing interest in the use of large language models, it is becoming increasingly important to understand whose views they express. These models tend to generate output that conforms to majority opinion and are not representative of diverse views. As a step toward building models that can take differing views into consideration, we build a novel corpus of social judgements. We crowdsourced annotations of a subset of the Commonsense Norm Bank that contained numbers in the situation descriptions and asked annotators to replace the number with a range defined by a start and end value that, in their view, correspond to the given verdict. Our corpus contains unaggregated annotations and annotator demographics. We describe our annotation process for social judgements and will release our dataset to support future work on numerical reasoning and perspectivist approaches to natural language processing.
Anthology ID:
2024.nlperspectives-1.4
Volume:
Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024
Month:
May
Year:
2024
Address:
Torino, Italia
Editors:
Gavin Abercrombie, Valerio Basile, Davide Bernadi, Shiran Dudy, Simona Frenda, Lucy Havens, Sara Tonelli
Venues:
NLPerspectives | WS
SIG:
Publisher:
ELRA and ICCL
Note:
Pages:
42–48
Language:
URL:
https://aclanthology.org/2024.nlperspectives-1.4
DOI:
Bibkey:
Cite (ACL):
Marlon May, Lucie Flek, and Charles Welch. 2024. A Perspectivist Corpus of Numbers in Social Judgements. In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024, pages 42–48, Torino, Italia. ELRA and ICCL.
Cite (Informal):
A Perspectivist Corpus of Numbers in Social Judgements (May et al., NLPerspectives-WS 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.nlperspectives-1.4.pdf