Vision is Not Enough: Enhancing Bounding Box Grounding for Document Key Information Extraction with OCR Anchors

Zhaochen Li, Rathika Suresh Kumar, Tao Bai, Zhijie Zhang, Hongxin Shao, Daniel Dahlmeier


Abstract
Vision Language Model (VLM)-based OCR-free pipelines are increasingly adopted for industrial Key Information Extraction (KIE) systems due to their simpler end-to-end architecture and reduced dependence on potentially noisy OCR outputs. However, their reliability in grounding extracted values to precise bounding boxes remains unclear, raising questions about their suitability for grounding-critical use cases. In this paper, we present a systematic study of bounding box grounding capability in both general and document-specific VLMs. Our findings reveal a consistent extraction-grounding gap across model families. We revisit the OCR system as a grounding primitive and find that delegating the grounding step to Large Language Models (LLMs), rather than VLMs, in a zero-shot manner substantially improves region-level grounding accuracy. We introduce a fine-tuned 0.6B LLM that further improves the predicted box alignment, achieving high overlap with the target field for in-domain data while maintaining low serving latency. These results position OCR-anchored grounding as a practical deployment choice for industrial KIE systems that require both extraction and precise localization.
Anthology ID:
2026.emnlp-industry.184
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Pepa Atanasova, Gholamreza Haffari, Minghan Wang
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
3114–3126
Language:
URL:
https://aclanthology.org/2026.emnlp-industry.184/
DOI:
Bibkey:
Cite (ACL):
Zhaochen Li, Rathika Suresh Kumar, Tao Bai, Zhijie Zhang, Hongxin Shao, and Daniel Dahlmeier. 2026. Vision is Not Enough: Enhancing Bounding Box Grounding for Document Key Information Extraction with OCR Anchors. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 3114–3126, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Vision is Not Enough: Enhancing Bounding Box Grounding for Document Key Information Extraction with OCR Anchors (Li et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-industry.184.pdf