%0 Conference Proceedings %T RoViST: Learning Robust Metrics for Visual Storytelling %A Wang, Eileen %A Han, Caren %A Poon, Josiah %Y Carpuat, Marine %Y de Marneffe, Marie-Catherine %Y Meza Ruiz, Ivan Vladimir %S Findings of the Association for Computational Linguistics: NAACL 2022 %D 2022 %8 July %I Association for Computational Linguistics %C Seattle, United States %F wang-etal-2022-rovist %X Visual storytelling (VST) is the task of generating a story paragraph that describes a given image sequence. Most existing storytelling approaches have evaluated their models using traditional natural language generation metrics like BLEU or CIDEr. However, such metrics based on n-gram matching tend to have poor correlation with human evaluation scores and do not explicitly consider other criteria necessary for storytelling such as sentence structure or topic coherence. Moreover, a single score is not enough to assess a story as it does not inform us about what specific errors were made by the model. In this paper, we propose 3 evaluation metrics sets that analyses which aspects we would look for in a good story: 1) visual grounding, 2) coherence, and 3) non-redundancy. We measure the reliability of our metric sets by analysing its correlation with human judgement scores on a sample of machine stories obtained from 4 state-of-the-arts models trained on the Visual Storytelling Dataset (VIST). Our metric sets outperforms other metrics on human correlation, and could be served as a learning based evaluation metric set that is complementary to existing rule-based metrics. %R 10.18653/v1/2022.findings-naacl.206 %U https://aclanthology.org/2022.findings-naacl.206 %U https://doi.org/10.18653/v1/2022.findings-naacl.206 %P 2691-2702