Comprehensive Multi-Modal Interactions for Referring Image Segmentation

Kanishk Jain, Vineet Gandhi


Abstract
We investigate Referring Image Segmentation (RIS), which outputs a segmentation map corresponding to the natural language description. Addressing RIS efficiently requires considering the interactions happening across visual and linguistic modalities and the interactions within each modality. Existing methods are limited because they either compute different forms of interactions sequentially (leading to error propagation) or ignore intra-modal interactions. We address this limitation by performing all three interactions simultaneously through a Synchronous Multi-Modal Fusion Module (SFM). Moreover, to produce refined segmentation masks, we propose a novel Hierarchical Cross-Modal Aggregation Module (HCAM), where linguistic features facilitate the exchange of contextual information across the visual hierarchy. We present thorough ablation studies and validate our approach’s performance on four benchmark datasets, showing considerable performance gains over the existing state-of-the-art (SOTA) methods.
Anthology ID:
2022.findings-acl.270
Volume:
Findings of the Association for Computational Linguistics: ACL 2022
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3427–3435
Language:
URL:
https://aclanthology.org/2022.findings-acl.270
DOI:
10.18653/v1/2022.findings-acl.270
Bibkey:
Cite (ACL):
Kanishk Jain and Vineet Gandhi. 2022. Comprehensive Multi-Modal Interactions for Referring Image Segmentation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3427–3435, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Comprehensive Multi-Modal Interactions for Referring Image Segmentation (Jain & Gandhi, Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-acl.270.pdf
Software:
 2022.findings-acl.270.software.zip
Video:
 https://aclanthology.org/2022.findings-acl.270.mp4
Code
 kanji95/SHNET
Data
Google RefexpMS COCORefCOCO