Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspective

Edoardo Manino, Julia Rozanova, Danilo Carvalho, Andre Freitas, Lucas Cordeiro


Abstract
Metamorphic testing has recently been used to check the safety of neural NLP models. Its main advantage is that it does not rely on a ground truth to generate test cases. However, existing studies are mostly concerned with robustness-like metamorphic relations, limiting the scope of linguistic properties they can test. We propose three new classes of metamorphic relations, which address the properties of systematicity, compositionality and transitivity. Unlike robustness, our relations are defined over multiple source inputs, thus increasing the number of test cases that we can produce by a polynomial factor. With them, we test the internal consistency of state-of-the-art NLP models, and show that they do not always behave according to their expected linguistic properties. Lastly, we introduce a novel graphical notation that efficiently summarises the inner structure of metamorphic relations.
Anthology ID:
2022.findings-acl.185
Volume:
Findings of the Association for Computational Linguistics: ACL 2022
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2355–2366
Language:
URL:
https://aclanthology.org/2022.findings-acl.185
DOI:
10.18653/v1/2022.findings-acl.185
Bibkey:
Cite (ACL):
Edoardo Manino, Julia Rozanova, Danilo Carvalho, Andre Freitas, and Lucas Cordeiro. 2022. Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspective. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2355–2366, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspective (Manino et al., Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-acl.185.pdf
Software:
 2022.findings-acl.185.software.zip
Video:
 https://aclanthology.org/2022.findings-acl.185.mp4