Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation

Yong Cao, Wei Li, Xianzhi Li, Min Chen, Guangyong Chen, Long Hu, Zhengdao Li, Kai Hwang


Abstract
Sign language recognition and translation first uses a recognition module to generate glosses from sign language videos and then employs a translation module to translate glosses into spoken sentences. Most existing works focus on the recognition step, while paying less attention to sign language translation. In this work, we propose a task-aware instruction network, namely TIN-SLT, for sign language translation, by introducing the isntruction module and the learning-based feature fuse strategy into a Transformer network. In this way, the pre-trained model’s language ability can be well explored and utilized to further boost the translation performance. Moreover, by exploring the representation space of sign language glosses and target spoken language, we propose a multi-level data augmentation scheme to adjust the data distribution of the training set. We conduct extensive experiments on two challenging benchmark datasets, PHOENIX-2014-T and ASLG-PC12, on which our method outperforms former best solutions by 1.65 and 1.42 in terms of BLEU-4. Our code and trained networks will be available upon the publication of this work.
Anthology ID:
2022.findings-naacl.205
Volume:
Findings of the Association for Computational Linguistics: NAACL 2022
Month:
July
Year:
2022
Address:
Seattle, United States
Editors:
Marine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2679–2690
Language:
URL:
https://aclanthology.org/2022.findings-naacl.205
DOI:
10.18653/v1/2022.findings-naacl.205
Bibkey:
Cite (ACL):
Yong Cao, Wei Li, Xianzhi Li, Min Chen, Guangyong Chen, Long Hu, Zhengdao Li, and Kai Hwang. 2022. Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2679–2690, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation (Cao et al., Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-naacl.205.pdf
Video:
 https://aclanthology.org/2022.findings-naacl.205.mp4
Code
 yongcaoplus/tin-slt