Interpretability for Morphological Inflection: from Character-level Predictions to Subword-level Rules

Tatyana Ruzsics, Olga Sozinova, Ximena Gutierrez-Vasques, Tanja Samardzic


Abstract
Neural models for morphological inflection have recently attained very high results. However, their interpretation remains challenging. Towards this goal, we propose a simple linguistically-motivated variant to the encoder-decoder model with attention. In our model, character-level cross-attention mechanism is complemented with a self-attention module over substrings of the input. We design a novel approach for pattern extraction from attention weights to interpret what the model learn. We apply our methodology to analyze the model’s decisions on three typologically-different languages and find that a) our pattern extraction method applied to cross-attention weights uncovers variation in form of inflection morphemes, b) pattern extraction from self-attention shows triggers for such variation, c) both types of patterns are closely aligned with grammar inflection classes and class assignment criteria, for all three languages. Additionally, we find that the proposed encoder attention component leads to consistent performance improvements over a strong baseline.
Anthology ID:
2021.eacl-main.278
Volume:
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
Month:
April
Year:
2021
Address:
Online
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3189–3201
Language:
URL:
https://aclanthology.org/2021.eacl-main.278
DOI:
10.18653/v1/2021.eacl-main.278
Bibkey:
Cite (ACL):
Tatyana Ruzsics, Olga Sozinova, Ximena Gutierrez-Vasques, and Tanja Samardzic. 2021. Interpretability for Morphological Inflection: from Character-level Predictions to Subword-level Rules. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3189–3201, Online. Association for Computational Linguistics.
Cite (Informal):
Interpretability for Morphological Inflection: from Character-level Predictions to Subword-level Rules (Ruzsics et al., EACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.eacl-main.278.pdf
Code
 tatyana-ruzsics/interpretable-inflection