Rˆ3Net:Relation-embedded Representation Reconstruction Network for Change Captioning

Yunbin Tu, Liang Li, Chenggang Yan, Shengxiang Gao, Zhengtao Yu


Abstract
Change captioning is to use a natural language sentence to describe the fine-grained disagreement between two similar images. Viewpoint change is the most typical distractor in this task, because it changes the scale and location of the objects and overwhelms the representation of real change. In this paper, we propose a Relation-embedded Representation Reconstruction Network (Rˆ3Net) to explicitly distinguish the real change from the large amount of clutter and irrelevant changes. Specifically, a relation-embedded module is first devised to explore potential changed objects in the large amount of clutter. Then, based on the semantic similarities of corresponding locations in the two images, a representation reconstruction module (RRM) is designed to learn the reconstruction representation and further model the difference representation. Besides, we introduce a syntactic skeleton predictor (SSP) to enhance the semantic interaction between change localization and caption generation. Extensive experiments show that the proposed method achieves the state-of-the-art results on two public datasets.
Anthology ID:
2021.emnlp-main.735
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
9319–9329
Language:
URL:
https://aclanthology.org/2021.emnlp-main.735
DOI:
10.18653/v1/2021.emnlp-main.735
Bibkey:
Cite (ACL):
Yunbin Tu, Liang Li, Chenggang Yan, Shengxiang Gao, and Zhengtao Yu. 2021. Rˆ3Net:Relation-embedded Representation Reconstruction Network for Change Captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9319–9329, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Rˆ3Net:Relation-embedded Representation Reconstruction Network for Change Captioning (Tu et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.735.pdf
Code
 tuyunbin/r3net
Data
Spot-the-diff