Scene Graph Modification as Incremental Structure Expanding

Xuming Hu, Zhijiang Guo, Yu Fu, Lijie Wen, Philip S. Yu


Abstract
A scene graph is a semantic representation that expresses the objects, attributes, and relationships between objects in a scene. Scene graphs play an important role in many cross modality tasks, as they are able to capture the interactions between images and texts. In this paper, we focus on scene graph modification (SGM), where the system is required to learn how to update an existing scene graph based on a natural language query. Unlike previous approaches that rebuilt the entire scene graph, we frame SGM as a graph expansion task by introducing the incremental structure expanding (ISE). ISE constructs the target graph by incrementally expanding the source graph without changing the unmodified structure. Based on ISE, we further propose a model that iterates between nodes prediction and edges prediction, inferring more accurate and harmonious expansion decisions progressively. In addition, we construct a challenging dataset that contains more complicated queries and larger scene graphs than existing datasets. Experiments on four benchmarks demonstrate the effectiveness of our approach, which surpasses the previous state-of-the-art model by large margins.
Anthology ID:
2022.coling-1.502
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
5707–5720
Language:
URL:
https://aclanthology.org/2022.coling-1.502
DOI:
Bibkey:
Cite (ACL):
Xuming Hu, Zhijiang Guo, Yu Fu, Lijie Wen, and Philip S. Yu. 2022. Scene Graph Modification as Incremental Structure Expanding. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5707–5720, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Scene Graph Modification as Incremental Structure Expanding (Hu et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.502.pdf
Data
COCORSICD