FiMA: Redundancy-Aware Description Enhancement for Gloss-Free Sign Language Translation

Peidong Liu, Lianyu Hu, Wei Feng


Abstract
Gloss-free Sign Language Translation (SLT) avoids costly gloss annotations but remains limited by weak visual-text alignment. Recent MLLM-based methods use frame descriptions as auxiliary supervision, yet these descriptions are often repetitive, producing nearly identical text features; meanwhile, global pooling washes out sparse but meaningful gestures. We propose FiMA (Fine-grained Multimodal Alignment), a pre-training framework for redundancy-aware description enhancement. FiMA combines a Description Enhancement Module, which removes shared template redundancy from description features, with a Multimodal Alignment Module, which aligns each text token to its most relevant visual frame through Max-Sim matching. On PHOENIX14T and CSL-Daily, FiMA achieves state-of-the-art gloss-free SLT performance and visualizations show clearer feature separation and temporal grounding. The code will be released at https://github.com/Ryeliu/FiMA.
Anthology ID:
2026.emnlp-main.2253
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
51247–51259
Language:
URL:
https://aclanthology.org/2026.emnlp-main.2253/
DOI:
Bibkey:
Cite (ACL):
Peidong Liu, Lianyu Hu, and Wei Feng. 2026. FiMA: Redundancy-Aware Description Enhancement for Gloss-Free Sign Language Translation. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 51247–51259, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
FiMA: Redundancy-Aware Description Enhancement for Gloss-Free Sign Language Translation (Liu et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.2253.pdf