Butterfly Effects in Frame Semantic Parsing: impact of data processing on model ranking

Alexandre Kabbach, Corentin Ribeyre, Aurélie Herbelot


Abstract
Knowing the state-of-the-art for a particular task is an essential component of any computational linguistics investigation. But can we be truly confident that the current state-of-the-art is indeed the best performing model? In this paper, we study the case of frame semantic parsing, a well-established task with multiple shared datasets. We show that in spite of all the care taken to provide a standard evaluation resource, small variations in data processing can have dramatic consequences for ranking parser performance. This leads us to propose an open-source standardized processing pipeline, which can be shared and reused for robust model comparison.
Anthology ID:
C18-1267
Volume:
Proceedings of the 27th International Conference on Computational Linguistics
Month:
August
Year:
2018
Address:
Santa Fe, New Mexico, USA
Editors:
Emily M. Bender, Leon Derczynski, Pierre Isabelle
Venue:
COLING
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3158–3169
Language:
URL:
https://aclanthology.org/C18-1267
DOI:
Bibkey:
Cite (ACL):
Alexandre Kabbach, Corentin Ribeyre, and Aurélie Herbelot. 2018. Butterfly Effects in Frame Semantic Parsing: impact of data processing on model ranking. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3158–3169, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Cite (Informal):
Butterfly Effects in Frame Semantic Parsing: impact of data processing on model ranking (Kabbach et al., COLING 2018)
Copy Citation:
PDF:
https://aclanthology.org/C18-1267.pdf
Code
 akb89/pyfn
Data
FrameNet