Efficient Pairwise Annotation of Argument Quality

Lukas Gienapp, Benno Stein, Matthias Hagen, Martin Potthast


Abstract
We present an efficient annotation framework for argument quality, a feature difficult to be measured reliably as per previous work. A stochastic transitivity model is combined with an effective sampling strategy to infer high-quality labels with low effort from crowdsourced pairwise judgments. The model’s capabilities are showcased by compiling Webis-ArgQuality-20, an argument quality corpus that comprises scores for rhetorical, logical, dialectical, and overall quality inferred from a total of 41,859 pairwise judgments among 1,271 arguments. With up to 93% cost savings, our approach significantly outperforms existing annotation procedures. Furthermore, novel insight into argument quality is provided through statistical analysis, and a new aggregation method to infer overall quality from individual quality dimensions is proposed.
Anthology ID:
2020.acl-main.511
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5772–5781
Language:
URL:
https://aclanthology.org/2020.acl-main.511
DOI:
10.18653/v1/2020.acl-main.511
Bibkey:
Cite (ACL):
Lukas Gienapp, Benno Stein, Matthias Hagen, and Martin Potthast. 2020. Efficient Pairwise Annotation of Argument Quality. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5772–5781, Online. Association for Computational Linguistics.
Cite (Informal):
Efficient Pairwise Annotation of Argument Quality (Gienapp et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-main.511.pdf
Video:
 http://slideslive.com/38928731