The American Sign Language (ASL) lexicon contains heterogeneous sign types, including established signed words, neologisms, ad-hoc depictions of real-world phenomena, and fingerspelled words. Productions within each type exhibit unique phonotactics and substantial variability, potentially confounding models pretrained on downstream ASL tasks. To survey the capacity for representation collapse, we isolate and annotate 14 types of ASL productions as they appear in science, technology, engineering, and math (STEM) educational content. We then probe seven pretrained models’ latent representations of the annotated clips, reporting collapse metrics with respect to domain, sign type, and meaning. Our findings show that out-of-distribution signs are more frequent in STEM content, and that pretrained models lose information about their visual and semantic contents in varing ways. Among the tested models, those which encoded the phonological properties of signs generalized to STEM data more effectively, while English- and gesture-based models can exhibit more brittleness. Future work can design interventions to mitigate the kinds of collapse documented here or extend this survey by evaluating additional models and datasets.