Tony Salomone
Author directory2026
Same Motions, Different Winners: The Gesture-Generation Leaderboard and Its Human Agreement Are Configuration-Tunable
Deep Gandhi | Ali Asaria | Tony Salomone
Proceedings of the 1st Workshop on Multimodal Interaction in Face-to-Face Dialogue
Deep Gandhi | Ali Asaria | Tony Salomone
Proceedings of the 1st Workshop on Multimodal Interaction in Face-to-Face Dialogue
Fréchet Gesture Distance (FGD) is the standard automatic metric for co-speech gesture generation, the models that synthesize the hand and body motion accompanying speech. Human perceptual studies remain the gold standard, and the GENEA challenges rank systems by them, but such studies are slow and costly, so papers routinely compare systems by FGD instead. FGD is a Fréchet distance in the latent space of a learned pose autoencoder, so it inherits the many choices behind that encoder, most of which papers never report. We ask whether FGD is a reliable proxy for perceptual evaluation under these unreported choices. Holding the systems fixed, we vary the reference tool’s own settings (analysis window, seed, feature versus raw space) and the corpus the feature extractor is trained on, on both tiers of the GENEA 2022 evaluation and on the GENEA 2023 challenge. First, the FGD ranking is not reproducible where it matters: the "best" system is one of two that a per-clip bootstrap shows are statistically tied, so which one wins is a configuration choice, though aggregate rank correlation stays moderate rather than collapsing. Second, and more consequential, FGD’s agreement with human ratings is modest and itself tunable, moving by 0.18 to 0.24 Kendall τ across these undocumented choices: FGD can be made to look appreciably more or less human-valid without changing the systems. These effects replicate across two challenge years and three evaluation tiers. We conclude that FGD numbers are not comparable across papers unless the window, seed, scoring space, and feature-extractor training corpus are pinned and reported, and that FGD cannot resolve small differences between closely performing systems.