Neuro-Symbolic Visual Dialog

Adnen Abdessaied, Mihai Bâce, Andreas Bulling


Abstract
We propose Neuro-Symbolic Visual Dialog (NSVD) —the first method to combine deep learning and symbolic program execution for multi-round visually-grounded reasoning. NSVD significantly outperforms existing purely-connectionist methods on two key challenges inherent to visual dialog: long-distance co-reference resolution as well as vanishing question-answering performance. We demonstrate the latter by proposing a more realistic and stricter evaluation scheme in which we use predicted answers for the full dialog history when calculating accuracy. We describe two variants of our model and show that using this new scheme, our best model achieves an accuracy of 99.72% on CLEVR-Dialog—a relative improvement of more than 10% over the state of the art—while only requiring a fraction of training data. Moreover, we demonstrate that our neuro-symbolic models have a higher mean first failure round, are more robust against incomplete dialog histories, and generalise better not only to dialogs that are up to three times longer than those seen during training but also to unseen question types and scenes.
Anthology ID:
2022.coling-1.17
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Editors:
Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, Seung-Hoon Na
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
192–217
Language:
URL:
https://aclanthology.org/2022.coling-1.17
DOI:
Bibkey:
Cite (ACL):
Adnen Abdessaied, Mihai Bâce, and Andreas Bulling. 2022. Neuro-Symbolic Visual Dialog. In Proceedings of the 29th International Conference on Computational Linguistics, pages 192–217, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Neuro-Symbolic Visual Dialog (Abdessaied et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.17.pdf
Code
 adnenabdessaied/NSVD
Data
CLEVRCLEVR-Dialog