Interactive Evidence Detection: train state-of-the-art model out-of-domain or simple model interactively?

Chris Stahlhut


Abstract
Finding evidence is of vital importance in research as well as fact checking and an evidence detection method would be useful in speeding up this process. However, when addressing a new topic there is no training data and there are two approaches to get started. One could use large amounts of out-of-domain data to train a state-of-the-art method, or to use the small data that a person creates while working on the topic. In this paper, we address this problem in two steps. First, by simulating users who read source documents and label sentences they can use as evidence, thereby creating small amounts of training data for an interactively trained evidence detection model; and second, by comparing such an interactively trained model against a pre-trained model that has been trained on large out-of-domain data. We found that an interactively trained model not only often out-performs a state-of-the-art model but also requires significantly lower amounts of computational resources. Therefore, especially when computational resources are scarce, e.g. no GPU available, training a smaller model on the fly is preferable to training a well generalising but resource hungry out-of-domain model.
Anthology ID:
D19-6613
Volume:
Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER)
Month:
November
Year:
2019
Address:
Hong Kong, China
Venue:
WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
79–89
Language:
URL:
https://aclanthology.org/D19-6613
DOI:
10.18653/v1/D19-6613
Bibkey:
Cite (ACL):
Chris Stahlhut. 2019. Interactive Evidence Detection: train state-of-the-art model out-of-domain or simple model interactively?. In Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER), pages 79–89, Hong Kong, China. Association for Computational Linguistics.
Cite (Informal):
Interactive Evidence Detection: train state-of-the-art model out-of-domain or simple model interactively? (Stahlhut, 2019)
Copy Citation:
PDF:
https://aclanthology.org/D19-6613.pdf
Attachment:
 D19-6613.Attachment.pdf