Exploring Methodologies for Collecting High-Quality Implicit Reasoning in Arguments

Keshav Singh, Farjana Sultana Mim, Naoya Inoue, Shoichi Naito, Kentaro Inui


Abstract
Annotation of implicit reasoning (i.e., warrant) in arguments is a critical resource to train models in gaining deeper understanding and correct interpretation of arguments. However, warrants are usually annotated in unstructured form, having no restriction on their lexical structure which sometimes makes it difficult to interpret how warrants relate to any of the information given in claim and premise. Moreover, assessing and determining better warrants from the large variety of reasoning patterns of unstructured warrants becomes a formidable task. Therefore, in order to annotate warrants in a more interpretative and restrictive way, we propose two methodologies to annotate warrants in a semi-structured form. To the best of our knowledge, we are the first to show how such semi-structured warrants can be annotated on a large scale via crowdsourcing. We demonstrate through extensive quality evaluation that our methodologies enable collecting better quality warrants in comparison to unstructured annotations. To further facilitate research towards the task of explicating warrants in arguments, we release our materials publicly (i.e., crowdsourcing guidelines and collected warrants).
Anthology ID:
2021.argmining-1.6
Volume:
Proceedings of the 8th Workshop on Argument Mining
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Editors:
Khalid Al-Khatib, Yufang Hou, Manfred Stede
Venue:
ArgMining
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
57–66
Language:
URL:
https://aclanthology.org/2021.argmining-1.6
DOI:
10.18653/v1/2021.argmining-1.6
Bibkey:
Cite (ACL):
Keshav Singh, Farjana Sultana Mim, Naoya Inoue, Shoichi Naito, and Kentaro Inui. 2021. Exploring Methodologies for Collecting High-Quality Implicit Reasoning in Arguments. In Proceedings of the 8th Workshop on Argument Mining, pages 57–66, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Exploring Methodologies for Collecting High-Quality Implicit Reasoning in Arguments (Singh et al., ArgMining 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.argmining-1.6.pdf
Code
 cl-tohoku/ukw-warrants