Identifying Annotator Bias: A new IRT-based method for bias identification

Jacopo Amidei, Paul Piwek, Alistair Willis


Abstract
A basic step in any annotation effort is the measurement of the Inter Annotator Agreement (IAA). An important factor that can affect the IAA is the presence of annotator bias. In this paper we introduce a new interpretation and application of the Item Response Theory (IRT) to detect annotators’ bias. Our interpretation of IRT offers an original bias identification method that can be used to compare annotators’ bias and characterise annotation disagreement. Our method can be used to spot outlier annotators, improve annotation guidelines and provide a better picture of the annotation reliability. Additionally, because scales for IAA interpretation are not generally agreed upon, our bias identification method is valuable as a complement to the IAA value which can help with understanding the annotation disagreement.
Anthology ID:
2020.coling-main.421
Volume:
Proceedings of the 28th International Conference on Computational Linguistics
Month:
December
Year:
2020
Address:
Barcelona, Spain (Online)
Editors:
Donia Scott, Nuria Bel, Chengqing Zong
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
4787–4797
Language:
URL:
https://aclanthology.org/2020.coling-main.421
DOI:
10.18653/v1/2020.coling-main.421
Bibkey:
Cite (ACL):
Jacopo Amidei, Paul Piwek, and Alistair Willis. 2020. Identifying Annotator Bias: A new IRT-based method for bias identification. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4787–4797, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Cite (Informal):
Identifying Annotator Bias: A new IRT-based method for bias identification (Amidei et al., COLING 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.coling-main.421.pdf