Machine Translation between Spoken Languages and Signed Languages Represented in SignWriting

Zifan Jiang, Amit Moryossef, Mathias Müller, Sarah Ebling


Abstract
This paper presents work on novel machine translation (MT) systems between spoken and signed languages, where signed languages are represented in SignWriting, a sign language writing system. Our work seeks to address the lack of out-of-the-box support for signed languages in current MT systems and is based on the SignBank dataset, which contains pairs of spoken language text and SignWriting content. We introduce novel methods to parse, factorize, decode, and evaluate SignWriting, leveraging ideas from neural factored MT. In a bilingual setup—translating from American Sign Language to (American) English—our method achieves over 30 BLEU, while in two multilingual setups—translating in both directions between spoken languages and signed languages—we achieve over 20 BLEU. We find that common MT techniques used to improve spoken language translation similarly affect the performance of sign language translation. These findings validate our use of an intermediate text representation for signed languages to include them in natural language processing research.
Anthology ID:
2023.findings-eacl.127
Volume:
Findings of the Association for Computational Linguistics: EACL 2023
Month:
May
Year:
2023
Address:
Dubrovnik, Croatia
Editors:
Andreas Vlachos, Isabelle Augenstein
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1706–1724
Language:
URL:
https://aclanthology.org/2023.findings-eacl.127
DOI:
10.18653/v1/2023.findings-eacl.127
Bibkey:
Cite (ACL):
Zifan Jiang, Amit Moryossef, Mathias Müller, and Sarah Ebling. 2023. Machine Translation between Spoken Languages and Signed Languages Represented in SignWriting. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1706–1724, Dubrovnik, Croatia. Association for Computational Linguistics.
Cite (Informal):
Machine Translation between Spoken Languages and Signed Languages Represented in SignWriting (Jiang et al., Findings 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.findings-eacl.127.pdf
Video:
 https://aclanthology.org/2023.findings-eacl.127.mp4