How do people talk about images? A study on open-domain conversations with images.

Yi-Pei Chen, Nobuyuki Shimizu, Takashi Miyazaki, Hideki Nakayama


Abstract
This paper explores how humans conduct conversations with images by investigating an open-domain image conversation dataset, ImageChat. We examined the conversations with images from the perspectives of image relevancy and image information. We found that utterances/conversations are not always related to the given image, and conversation topics diverge within three turns about half of the time. Besides image objects, more comprehensive non-object image information is also indispensable. After inspecting the causes, we suggested that understanding the overall scenario of image and connecting objects based on their high-level attributes might be very helpful to generate more engaging open-domain conversations when an image is presented. We proposed enriching the image information with image caption and object tags based on our analysis. With our proposed image+ features, we improved automatic metrics including BLEU and Bert Score, and increased the diversity and image-relevancy of generated responses to the strong baseline. The result verifies that our analysis provides valuable insights and could facilitate future research on open-domain conversations with images.
Anthology ID:
2022.naacl-srw.20
Volume:
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop
Month:
July
Year:
2022
Address:
Hybrid: Seattle, Washington + Online
Editors:
Daphne Ippolito, Liunian Harold Li, Maria Leonor Pacheco, Danqi Chen, Nianwen Xue
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
156–162
Language:
URL:
https://aclanthology.org/2022.naacl-srw.20
DOI:
10.18653/v1/2022.naacl-srw.20
Bibkey:
Cite (ACL):
Yi-Pei Chen, Nobuyuki Shimizu, Takashi Miyazaki, and Hideki Nakayama. 2022. How do people talk about images? A study on open-domain conversations with images.. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop, pages 156–162, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics.
Cite (Informal):
How do people talk about images? A study on open-domain conversations with images. (Chen et al., NAACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.naacl-srw.20.pdf
Video:
 https://aclanthology.org/2022.naacl-srw.20.mp4
Data
Image-Chat