Semantic overfitting: what ‘world’ do we consider when evaluating disambiguation of text?

Filip Ilievski, Marten Postma, Piek Vossen


Abstract
Semantic text processing faces the challenge of defining the relation between lexical expressions and the world to which they make reference within a period of time. It is unclear whether the current test sets used to evaluate disambiguation tasks are representative for the full complexity considering this time-anchored relation, resulting in semantic overfitting to a specific period and the frequent phenomena within. We conceptualize and formalize a set of metrics which evaluate this complexity of datasets. We provide evidence for their applicability on five different disambiguation tasks. To challenge semantic overfitting of disambiguation systems, we propose a time-based, metric-aware method for developing datasets in a systematic and semi-automated manner, as well as an event-based QA task.
Anthology ID:
C16-1112
Volume:
Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers
Month:
December
Year:
2016
Address:
Osaka, Japan
Editors:
Yuji Matsumoto, Rashmi Prasad
Venue:
COLING
SIG:
Publisher:
The COLING 2016 Organizing Committee
Note:
Pages:
1180–1191
Language:
URL:
https://aclanthology.org/C16-1112
DOI:
Bibkey:
Cite (ACL):
Filip Ilievski, Marten Postma, and Piek Vossen. 2016. Semantic overfitting: what ‘world’ do we consider when evaluating disambiguation of text?. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1180–1191, Osaka, Japan. The COLING 2016 Organizing Committee.
Cite (Informal):
Semantic overfitting: what ‘world’ do we consider when evaluating disambiguation of text? (Ilievski et al., COLING 2016)
Copy Citation:
PDF:
https://aclanthology.org/C16-1112.pdf
Data
AMR BankECB+