Argus-Retriever: Vision-LLM Late-Interaction Retrieval with Region-Aware Query-Conditioned MoE for Visual Document Retrieval

Abdelrahman Abdallah, Mahmoud Abdalla, Mohammed Ali, Adam Jatowt


Abstract
Late-interaction vision-language retrievers represent each document page as many visual token embeddings and score queries with MaxSim. In systems such as ColPali, ColQwen, ColNomic, and Nemotron ColEmbed, the document embeddings are produced without seeing the query, so the same page is represented identically for a table lookup, a chart question, and a layout-sensitive evidence request. We introduce Argus, a family of query-conditioned late-interaction retrievers built on Qwen3.5-VL. Argus adds a region-aware Mixture-of-Experts module: the query encoder produces both retrieval embeddings and a compact context vector, the document page is pooled into spatial regions, and a query-aware router selects latent experts per region before MaxSim. The output remains a multi-vector index compatible with ColPali-style retrieval, but the document representation is now dependent on the query (i.e., 𝐃(q)). All Argus models use a 1024-dimensional retrieval head, compared with the 2560-dimensional and 4096-dimensional heads of recent state-of-the-art systems, and are trained on roughly 9% of the available public supervision rather than the full pool. The 9B model reaches 92.67 NDCG@5 on ViDoRe V1 and 86.0 NDCG@5 on the combined V1+V2 leaderboard, the highest reported value for an open late-interaction model on the combined leaderboard. Wrapped in a Qwen3.6-27B agentic retrieval pipeline on ViDoRe V3, Argus-9B further improves its NDCG@10 from 60.28 to 64.80 over public tasks, showing that the same retriever serves both as a strong standalone system and as a search primitive for iterative LLM agents.
Anthology ID:
2026.emnlp-main.352
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
8075–8094
Language:
URL:
https://aclanthology.org/2026.emnlp-main.352/
DOI:
Bibkey:
Cite (ACL):
Abdelrahman Abdallah, Mahmoud Abdalla, Mohammed Ali, and Adam Jatowt. 2026. Argus-Retriever: Vision-LLM Late-Interaction Retrieval with Region-Aware Query-Conditioned MoE for Visual Document Retrieval. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 8075–8094, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Argus-Retriever: Vision-LLM Late-Interaction Retrieval with Region-Aware Query-Conditioned MoE for Visual Document Retrieval (Abdallah et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.352.pdf