Stance-Aware Re-Ranking for Non-factual Comparative Queries

Jan Heinrich Reimer, Alexander Bondarenko, Maik Fröbe, Matthias Hagen


Abstract
We propose a re-ranking approach to improve the retrieval effectiveness for non-factual comparative queries like ‘Which city is better, London or Paris?’ based on whether the results express a stance towards the comparison objects (London vs. Paris) or not. Applied to the 26 runs submitted to the Touché 2022 task on comparative argument retrieval, our stance-aware re-ranking significantly improves the retrieval effectiveness for all runs when perfect oracle-style stance labels are available. With our most effective practical stance detector based on GPT-3.5 (F₁ of 0.49 on four stance classes), our re-ranking still improves the effectiveness for all runs but only six improvements are significant. Artificially “deteriorating” the oracle-style labels, we further find that an F₁ of 0.90 for stance detection is necessary to significantly improve the retrieval effectiveness for the best run via stance-aware re-ranking.
Anthology ID:
2023.argmining-1.5
Volume:
Proceedings of the 10th Workshop on Argument Mining
Month:
December
Year:
2023
Address:
Singapore
Editors:
Milad Alshomary, Chung-Chi Chen, Smaranda Muresan, Joonsuk Park, Julia Romberg
Venues:
ArgMining | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
45–51
Language:
URL:
https://aclanthology.org/2023.argmining-1.5
DOI:
10.18653/v1/2023.argmining-1.5
Bibkey:
Cite (ACL):
Jan Heinrich Reimer, Alexander Bondarenko, Maik Fröbe, and Matthias Hagen. 2023. Stance-Aware Re-Ranking for Non-factual Comparative Queries. In Proceedings of the 10th Workshop on Argument Mining, pages 45–51, Singapore. Association for Computational Linguistics.
Cite (Informal):
Stance-Aware Re-Ranking for Non-factual Comparative Queries (Reimer et al., ArgMining-WS 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.argmining-1.5.pdf
Software:
 2023.argmining-1.5.Software.zip