A Graph Autoencoder Approach for Gesture Classification with Gesture AMR

Huma Jamil, Ibrahim Khebour, Kenneth Lai, James Pustejovsky, Nikhil Krishnaswamy


Abstract
We present a novel graph autoencoder (GAE) architecture for classifying gestures using Gesture Abstract Meaning Representation (GAMR), a structured semantic annotation framework for gestures in collaborative tasks. We leverage the inherent graphical structure of GAMR by employing Graph Neural Networks (GNNs), specifically an Edge-aware Graph Attention Network (EdgeGAT), to learn embeddings of gesture semantic representations. Using the EGGNOG dataset, which captures diverse physical gesture forms expressing similar semantics, we evaluate our GAE on a multi-label classification task for gestural actions. Results indicate that our approach significantly outperforms naive baselines and is competitive with specialized Transformer-based models like AMRBART, despite using considerably fewer parameters and no pretraining. This work highlights the effectiveness of structured graphical representations in modeling multimodal semantics, offering a scalable and efficient approach to gesture interpretation in situated human-agent collaborative scenarios.
Anthology ID:
2025.iwcs-main.4
Volume:
Proceedings of the 16th International Conference on Computational Semantics
Month:
September
Year:
2025
Address:
Düsseldorf, Germany
Editors:
Kilian Evang, Laura Kallmeyer, Sylvain Pogodalla
Venue:
IWCS
SIG:
SIGSEM
Publisher:
Association for Computational Linguistics
Note:
Pages:
41–48
Language:
URL:
https://aclanthology.org/2025.iwcs-main.4/
DOI:
Bibkey:
Cite (ACL):
Huma Jamil, Ibrahim Khebour, Kenneth Lai, James Pustejovsky, and Nikhil Krishnaswamy. 2025. A Graph Autoencoder Approach for Gesture Classification with Gesture AMR. In Proceedings of the 16th International Conference on Computational Semantics, pages 41–48, Düsseldorf, Germany. Association for Computational Linguistics.
Cite (Informal):
A Graph Autoencoder Approach for Gesture Classification with Gesture AMR (Jamil et al., IWCS 2025)
Copy Citation:
PDF:
https://aclanthology.org/2025.iwcs-main.4.pdf