Ranking Passages for Argument Convincingness

Peter Potash, Adam Ferguson, Timothy J. Hazen


Abstract
In data ranking applications, pairwise annotation is often more consistent than cardinal annotation for learning ranking models. We examine this in a case study on ranking text passages for argument convincingness. Our task is to choose text passages that provide the highest-quality, most-convincing arguments for opposing sides of a topic. Using data from a deployed system within the Bing search engine, we construct a pairwise-labeled dataset for argument convincingness that is substantially more comprehensive in topical coverage compared to existing public resources. We detail the process of extracting topical passages for queries submitted to a search engine, creating annotated sets of passages aligned to different stances on a topic, and assessing argument convincingness of passages using pairwise annotation. Using a state-of-the-art convincingness model, we evaluate several methods for using pairwise-annotated data examples to train models for ranking passages. Our results show pairwise training outperforms training that regresses to a target score for each passage. Our results also show a simple ‘win-rate’ score is a better regression target than the previously proposed page-rank target. Lastly, addressing the need to filter noisy crowd-sourced annotations when constructing a dataset, we show that filtering for transitivity within pairwise annotations is more effective than filtering based on annotation confidence measures for individual examples.
Anthology ID:
W19-4517
Volume:
Proceedings of the 6th Workshop on Argument Mining
Month:
August
Year:
2019
Address:
Florence, Italy
Editors:
Benno Stein, Henning Wachsmuth
Venue:
ArgMining
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
146–155
Language:
URL:
https://aclanthology.org/W19-4517
DOI:
10.18653/v1/W19-4517
Bibkey:
Cite (ACL):
Peter Potash, Adam Ferguson, and Timothy J. Hazen. 2019. Ranking Passages for Argument Convincingness. In Proceedings of the 6th Workshop on Argument Mining, pages 146–155, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
Ranking Passages for Argument Convincingness (Potash et al., ArgMining 2019)
Copy Citation:
PDF:
https://aclanthology.org/W19-4517.pdf