ChartREG++: Benchmarking and Improving Chart Referring Expression Grounding under Diverse referring clues and Multi-Target Referring

Tianhao Niu, Ziyu Han, Xuan Dong, Qingfu Zhu, Wanxiang Che


Abstract
Referring expression grounding is a core problem in visual grounding and is widely used as a diagnostic of spatial grounding and reasoning in vision–language models, yet most prior work focuses on natural images. In contrast, existing chart referring expression grounding-related benchmarks remain limited: (1) they largely adopt bounding boxes, constraining localization precision for fine chart elements; (2) they mostly assume one and two referred target instances, failing to handle multi-instance target references; (3) the language expressions over-rely on textual cues or data-ranking clues in the chart; (4) they cover only a narrow range of chart types. To address these issues, we introduce a chart referring expression grounding benchmark that first systematically supports multiple localization formats, multiple referred targets, diverse grounding cues and diverse chart types. Results across representative multi-modal large models reveal a significant performance gap. We further introduce an LLM-free synthesis pipeline that exploits the inherent alignment between matplotlib internal artist hierarchy and rendered chart primitives to derive pixel-accurate, multi-granularity instance segmentation masks. We train an instance segmentation model with the synthesized masks and integrate it into a general-purpose multi-modal grounding framework. The resulting system consistently outperforms baselines on our benchmark and generalizes well to a ChartQA-derived real-chart grounding benchmark.
Anthology ID:
2026.emnlp-main.1485
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
33945–33969
Language:
URL:
https://aclanthology.org/2026.emnlp-main.1485/
DOI:
Bibkey:
Cite (ACL):
Tianhao Niu, Ziyu Han, Xuan Dong, Qingfu Zhu, and Wanxiang Che. 2026. ChartREG++: Benchmarking and Improving Chart Referring Expression Grounding under Diverse referring clues and Multi-Target Referring. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 33945–33969, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
ChartREG++: Benchmarking and Improving Chart Referring Expression Grounding under Diverse referring clues and Multi-Target Referring (Niu et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.1485.pdf