Experimental Machine Translation of the Swiss German Sign Language via 3D Augmentation of Body Keypoints

Lorenz Hufe, Eleftherios Avramidis


Abstract
This paper describes the participation of DFKI-SLT at the Sign Language Translation Task of the Seventh Conference of Machine Translation (WMT22). The system focuses on the translation direction from the Swiss German Sign Language (DSGS) to written German. The original videos of the sign language were analyzed with computer vision models to provide 3D body keypoints. A deep-learning sequence-to-sequence model is trained on a parallel corpus of these body keypoints aligned to written German sentences. Geometric data augmentation occurs during the training process. The body keypoints are augmented by artificial rotation in the three dimensional space. The 3D-transformation is calculated with different angles on every batch of the training process.
Anthology ID:
2022.wmt-1.95
Volume:
Proceedings of the Seventh Conference on Machine Translation (WMT)
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates (Hybrid)
Editors:
Philipp Koehn, Loïc Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, Marcos Zampieri
Venue:
WMT
SIG:
SIGMT
Publisher:
Association for Computational Linguistics
Note:
Pages:
983–988
Language:
URL:
https://aclanthology.org/2022.wmt-1.95
DOI:
Bibkey:
Cite (ACL):
Lorenz Hufe and Eleftherios Avramidis. 2022. Experimental Machine Translation of the Swiss German Sign Language via 3D Augmentation of Body Keypoints. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 983–988, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Cite (Informal):
Experimental Machine Translation of the Swiss German Sign Language via 3D Augmentation of Body Keypoints (Hufe & Avramidis, WMT 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.wmt-1.95.pdf