Matteo Lanza
Author directory2026
Using Sign Phonology to Improve Few-Shot Cross-Lingual Transfer Learning for Sign Language Recognition
Matteo Lanza
Proceedings of the Second Workshop on Sign Language Processing
Matteo Lanza
Proceedings of the Second Workshop on Sign Language Processing
Pose-based sign language recognizers represent the two hands as separate halves of a skeleton graph, so a left-dominant and a right-dominant signer produce the same sign as mirror images, and a model must learn every sign in both orientations. Which hand leads, however, is a fact about the signer, not the sign. We therefore canonicalize dominance, mapping the hands onto sign phonology’s dominant/non-dominant roles: any training clip whose left hand carries clearly more motion energy is mirrored – a lossless operation, since the mirror is an exact automorphism of the skeleton. Canonicalized donors trained on Turkish Sign Language transfer to six other sign languages +6.01 accuracy percentage points better than standard donors under full fine-tuning, and +5.19 with frozen features; every canonicalized donor outscores every standard one (exact p = 1/70 over n = 8 donors), the gain is positive on all six recipients, and most of the gain remains even when recipient data is not canonicalized. To locate the source of the gain, we train nine donor-training variants that contain identical data and differ only in which clips are mirrored. Every other variant beats the natural donor, and three that keep the natural left/right proportion recover 57 to 79% of the gain (so a global orientation change is not the whole effect); this narrows the mechanism without identifying it. One variant, a random 20% re-mirroring after canonicalization, outscores the operator itself, so it is sufficient rather than optimal.
The motion-energy comparison itself is fragile: across seven corpora, 0.12–65.1% of clips never have a hand detected, so up to 80.7% of apparently one-handed clips are tracking artifacts rather than signing. Restricting the transfer comparison to clips in which both hands were tracked shrinks the fine-tune gain from +5.72 to +4.77 with full separation (exact p = 1/15 using four standard and two canonical donors). We also release relevant per-corpus validity checks.
The motion-energy comparison itself is fragile: across seven corpora, 0.12–65.1% of clips never have a hand detected, so up to 80.7% of apparently one-handed clips are tracking artifacts rather than signing. Restricting the transfer comparison to clips in which both hands were tracked shrinks the fine-tune gain from +5.72 to +4.77 with full separation (exact p = 1/15 using four standard and two canonical donors). We also release relevant per-corpus validity checks.