Ensemble ALBERT and RoBERTa for Span Prediction in Question Answering

Sony Bachina, Spandana Balumuri, Sowmya Kamath S


Abstract
Retrieving relevant answers from heterogeneous data formats, for given for questions, is a challenging problem. The process of pinpointing relevant information suitable to answer a question is further compounded in large document collections containing documents of substantial length. This paper presents the models designed as part of our submission to the DialDoc21 Shared Task (Document-grounded Dialogue and Conversational Question Answering) for span prediction in question answering. The proposed models leverage the superior predictive power of pretrained transformer models like RoBERTa, ALBERT and ELECTRA, to identify the most relevant information in an associated passage for the next agent turn. To further enhance the performance, the models were fine-tuned on different span selection based question answering datasets like SQuAD2.0 and Natural Questions (NQ) corpus. We also explored ensemble techniques for combining multiple models to achieve enhanced performance for the task. Our team SB_NITK ranked 6th on the leaderboard for the Knowledge Identification task, and our best ensemble model achieved an Exact score of 58.58 and an F1 score of 73.39.
Anthology ID:
2021.dialdoc-1.9
Volume:
Proceedings of the 1st Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc 2021)
Month:
August
Year:
2021
Address:
Online
Venue:
dialdoc
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
63–68
Language:
URL:
https://aclanthology.org/2021.dialdoc-1.9
DOI:
10.18653/v1/2021.dialdoc-1.9
Bibkey:
Cite (ACL):
Sony Bachina, Spandana Balumuri, and Sowmya Kamath S. 2021. Ensemble ALBERT and RoBERTa for Span Prediction in Question Answering. In Proceedings of the 1st Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc 2021), pages 63–68, Online. Association for Computational Linguistics.
Cite (Informal):
Ensemble ALBERT and RoBERTa for Span Prediction in Question Answering (Bachina et al., dialdoc 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.dialdoc-1.9.pdf
Data
CORD-19Natural QuestionsSQuAD