Modeling Intensification for Sign Language Generation: A Computational Approach

Mert Inan, Yang Zhong, Sabit Hassan, Lorna Quandt, Malihe Alikhani


Abstract
End-to-end sign language generation models do not accurately represent the prosody in sign language. A lack of temporal and spatial variations leads to poor-quality generated presentations that confuse human interpreters. In this paper, we aim to improve the prosody in generated sign languages by modeling intensification in a data-driven manner. We present different strategies grounded in linguistics of sign language that inform how intensity modifiers can be represented in gloss annotations. To employ our strategies, we first annotate a subset of the benchmark PHOENIX-14T, a German Sign Language dataset, with different levels of intensification. We then use a supervised intensity tagger to extend the annotated dataset and obtain labels for the remaining portion of it. This enhanced dataset is then used to train state-of-the-art transformer models for sign language generation. We find that our efforts in intensification modeling yield better results when evaluated with automatic metrics. Human evaluation also indicates a higher preference of the videos generated using our model.
Anthology ID:
2022.findings-acl.228
Volume:
Findings of the Association for Computational Linguistics: ACL 2022
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2897–2911
Language:
URL:
https://aclanthology.org/2022.findings-acl.228
DOI:
10.18653/v1/2022.findings-acl.228
Bibkey:
Cite (ACL):
Mert Inan, Yang Zhong, Sabit Hassan, Lorna Quandt, and Malihe Alikhani. 2022. Modeling Intensification for Sign Language Generation: A Computational Approach. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2897–2911, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Modeling Intensification for Sign Language Generation: A Computational Approach (Inan et al., Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-acl.228.pdf
Software:
 2022.findings-acl.228.software.zip
Video:
 https://aclanthology.org/2022.findings-acl.228.mp4
Code
 merterm/modeling-intensification-for-slg
Data
PHOENIX14T