Sihan Tan
2025
Improvement in Sign Language Translation Using Text CTC Alignment
Sihan Tan
|
Taro Miyazaki
|
Nabeela Khan
|
Kazuhiro Nakadai
Proceedings of the 31st International Conference on Computational Linguistics
Current sign language translation (SLT) approaches often rely on gloss-based supervision with Connectionist Temporal Classification (CTC), limiting their ability to handle non-monotonic alignments between sign language video and spoken text. In this work, we propose a novel method combining joint CTC/Attention and transfer learning. The joint CTC/Attention introduces hierarchical encoding and integrates CTC with the attention mechanism during decoding, effectively managing both monotonic and non-monotonic alignments. Meanwhile, transfer learning helps bridge the modality gap between vision and language in SLT. Experimental results on two widely adopted benchmarks, RWTH-PHOENIX-Weather 2014 T and CSL-Daily, show that our method achieves results comparable to state-of-the-art and outperforms the pure-attention baseline. Additionally, this work opens a new door for future research into gloss-free SLT using text-based CTC alignment.
2024
Sign Language Translation with Gloss Pair Encoding
Taro Miyazaki
|
Sihan Tan
|
Tsubasa Uchida
|
Hiroyuki Kaneko
Proceedings of the LREC-COLING 2024 11th Workshop on the Representation and Processing of Sign Languages: Evaluation of Sign Language Resources
SEDA: Simple and Effective Data Augmentation for Sign Language Understanding
Sihan Tan
|
Taro Miyazaki
|
Katsutoshi Itoyama
|
Kazuhiro Nakadai
Proceedings of the LREC-COLING 2024 11th Workshop on the Representation and Processing of Sign Languages: Evaluation of Sign Language Resources
Search
Fix data
Co-authors
- Taro Miyazaki 3
- Kazuhiro Nakadai 2
- Katsutoshi Itoyama 1
- Hiroyuki Kaneko 1
- Nabeela Khan 1
- show all...