SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge

Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, Manfred Pinkal


Abstract
This report summarizes the results of the SemEval 2018 task on machine comprehension using commonsense knowledge. For this machine comprehension task, we created a new corpus, MCScript. It contains a high number of questions that require commonsense knowledge for finding the correct answer. 11 teams from 4 different countries participated in this shared task, most of them used neural approaches. The best performing system achieves an accuracy of 83.95%, outperforming the baselines by a large margin, but still far from the human upper bound, which was found to be at 98%.
Anthology ID:
S18-1119
Volume:
Proceedings of the 12th International Workshop on Semantic Evaluation
Month:
June
Year:
2018
Address:
New Orleans, Louisiana
Editors:
Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, Marine Carpuat
Venue:
SemEval
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
747–757
Language:
URL:
https://aclanthology.org/S18-1119
DOI:
10.18653/v1/S18-1119
Bibkey:
Cite (ACL):
Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. 2018. SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 747–757, New Orleans, Louisiana. Association for Computational Linguistics.
Cite (Informal):
SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge (Ostermann et al., SemEval 2018)
Copy Citation:
PDF:
https://aclanthology.org/S18-1119.pdf
Data
ConceptNetMCScriptMCTestNewsQARACESQuADTriviaQA