Phonological similarity is fundamental to sign language structure and cross-linguistic analysis, yet existing approaches based on HamNoSys operate only at the transcription level and ignore visual realization. We propose a phonology-aware model that predicts continuous phonological similarity directly from sign videos while remaining grounded in symbolic phonological structure. Our landmark-based approach uses body and hand keypoints extracted with MediaPipe and a temporal encoder combining short-term convolution with Head-Token Mixing Attention to capture both local articulation and long-range dynamics. By factorizing HamNoSys annotations into interpretable phonological components and using them as auxiliary supervision, the model learns robust and interpretable representations. Experiments on four sign languages show consistent improvements over alignment-based and RGB-based baselines, particularly in cross-linguistic generalization.