Neural Conversational QA: Learning to Reason vs Exploiting Patterns

Nikhil Verma, Abhishek Sharma, Dhiraj Madan, Danish Contractor, Harshit Kumar, Sachindra Joshi


Abstract
Neural Conversational QA tasks such as ShARC require systems to answer questions based on the contents of a given passage. On studying recent state-of-the-art models on the ShARC QA task, we found indications that the model(s) learn spurious clues/patterns in the data-set. Further, a heuristic-based program, built to exploit these patterns, had comparative performance to that of the neural models. In this paper we share our findings about the four types of patterns in the ShARC corpus and how the neural models exploit them. Motivated by the above findings, we create and share a modified data-set that has fewer spurious patterns than the original data-set, consequently allowing models to learn better.
Anthology ID:
2020.emnlp-main.589
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7263–7269
Language:
URL:
https://aclanthology.org/2020.emnlp-main.589
DOI:
10.18653/v1/2020.emnlp-main.589
Bibkey:
Cite (ACL):
Nikhil Verma, Abhishek Sharma, Dhiraj Madan, Danish Contractor, Harshit Kumar, and Sachindra Joshi. 2020. Neural Conversational QA: Learning to Reason vs Exploiting Patterns. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7263–7269, Online. Association for Computational Linguistics.
Cite (Informal):
Neural Conversational QA: Learning to Reason vs Exploiting Patterns (Verma et al., EMNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.589.pdf
Video:
 https://slideslive.com/38939347
Data
Neural Conversational QA