Can a Transformer Pass the Wug Test? Tuning Copying Bias in Neural Morphological Inflection Models

Ling Liu, Mans Hulden


Abstract
Deep learning sequence models have been successful with morphological inflection generation. The SIGMORPHON shared task results in the past several years indicate that such models can perform well, but only if the training data covers a good amount of different lemmata, or if the lemmata to be inflected at test time have also been seen in training, as has indeed been largely the case in these tasks. Surprisingly, we find that standard models such as the Transformer almost completely fail at generalizing inflection patterns when trained on a limited number of lemmata and asked to inflect previously unseen lemmata—i.e. under “wug test”-like circumstances. This is true even though the actual number of training examples is very large. While established data augmentation techniques can be employed to alleviate this shortcoming by introducing a copying bias through hallucinating synthetic new word forms using the alphabet in the language at hand, our experiment results show that, to be more effective, the hallucination process needs to pay attention to substrings of syllable-like length rather than individual characters.
Anthology ID:
2022.acl-short.84
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
739–749
Language:
URL:
https://aclanthology.org/2022.acl-short.84
DOI:
10.18653/v1/2022.acl-short.84
Bibkey:
Cite (ACL):
Ling Liu and Mans Hulden. 2022. Can a Transformer Pass the Wug Test? Tuning Copying Bias in Neural Morphological Inflection Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 739–749, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Can a Transformer Pass the Wug Test? Tuning Copying Bias in Neural Morphological Inflection Models (Liu & Hulden, ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-short.84.pdf
Software:
 2022.acl-short.84.software.zip