Chen Bo Calvin Zhang
Author directory2026
Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning
Xingang Guo | Utkarsh Tyagi | Advait Gosai | Paula Vergara | Jayeon Park | Ernesto Gabriel Hernández Montoya | Chen Bo Calvin Zhang | Bin Hu | Yunzhong He | Bing Liu | Rakshith Sharma Srinivasa
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Xingang Guo | Utkarsh Tyagi | Advait Gosai | Paula Vergara | Jayeon Park | Ernesto Gabriel Hernández Montoya | Chen Bo Calvin Zhang | Bin Hu | Yunzhong He | Bing Liu | Rakshith Sharma Srinivasa
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Think-with-images, where multimodal LLMs (MLLMs) actively manipulate visual inputs as part of their reasoning, has become a focal research direction for agentic multimodal systems, with a surge of prompting, supervised, and reinforcement learning (RL) methods teaching models to invoke vision tools such as cropping, zooming, and editing. Yet most established multimodal benchmarks use images whose critical content is already easy to perceive in a single pass, and many have saturated on frontier models. We introduce VisualToolBench (VTB), a benchmark deliberately curated so that critical visual content is dense, occluded, dispersed, rotated, or low-contrast, making active image manipulation a prerequisite for success. VTB contains 1,204 expert-authored, open-ended tasks (603 single-turn, 601 multi-turn) across five domains. Each task is paired with rubrics tagged by five competency categories (Visual Understanding, Truthfulness, Instruction Following, Reasoning, and Writing Style) that enable per-axis disentanglement. Across 26 evaluated MLLMs, even the strongest model reaches only 29% accuracy, with Visual Understanding the consistent bottleneck and visual misinterpretation accounting for 32% to 39% of failures. Top models, moreover, converge to comparable accuracy through markedly different tool-use strategies, which we analyze in detail.
PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning
Afra Feyza Akyürek | Advait Gosai | Chen Bo Calvin Zhang | Vipul Gupta | Jaehwan Jeong | Anisha Gunjal | Tahseen Rabbani | Maria Mazzone | David Randolph IV | Mohammad Mahmoudi Meymand | Gurshaan Chattha | Paula Rodriguez | Diego A. Mares Buendia | Pavit Singh | Michael Liu | Subodh Chawla | Peter Cline | Lucy Ogaz | Ernesto Gabriel Hernández Montoya | Zihao Wang | Pavi Bhatter | Marcos Ayestaran | Bing Liu | Yunzhong He
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Afra Feyza Akyürek | Advait Gosai | Chen Bo Calvin Zhang | Vipul Gupta | Jaehwan Jeong | Anisha Gunjal | Tahseen Rabbani | Maria Mazzone | David Randolph IV | Mohammad Mahmoudi Meymand | Gurshaan Chattha | Paula Rodriguez | Diego A. Mares Buendia | Pavit Singh | Michael Liu | Subodh Chawla | Peter Cline | Lucy Ogaz | Ernesto Gabriel Hernández Montoya | Zihao Wang | Pavi Bhatter | Marcos Ayestaran | Bing Liu | Yunzhong He
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Frontier model progress is often measured using academic benchmarks that provide a limited view of performance on open-ended, economically consequential tasks in high-stakes professional domains where practical returns matter most. We introduce Professional Reasoning Bench (PRBench), a realistic, open-ended, and difficult benchmark of real-world problems in Finance and Law. We open-source its 1,100 expert-authored tasks and 19,356 expert-curated criteria, making it the largest public, rubric-based benchmark for both legal and finance domains. We recruit 182 qualified professionals, holding JDs, CFAs, or 6+ years of experience, who contributed questions inspired by their actual workflows. This process yields significant diversity, with tasks spanning 114 countries and 47 US jurisdictions. Our expert-curated rubrics are validated through a rigorous quality pipeline, including independent expert validation. Subsequent evaluation of 20 leading models reveals substantial room for improvement, with top scores of only 0.39 (Finance) and 0.37 (Legal) on our Hard subsets. We further catalog associated economic impacts of the prompts and analyze performance using human-annotated rubric categories. Common failure modes include inaccurate judgments, a lack of process transparency and incomplete reasoning, highlighting critical gaps in their reliability for professional adoption.
Search
Fix author
Co-authors
- Advait Gosai 2
- Yunzhong He 2
- Bing Liu 2
- Ernesto Gabriel Hernández Montoya 2
- Afra Feyza Akyürek 1
- Marcos Ayestaran 1
- Pavi Bhatter 1
- Diego A. Mares Buendia 1
- Gurshaan Chattha 1
- Subodh Chawla 1
- Peter Cline 1
- Anisha Gunjal 1
- Xingang Guo 1
- Vipul Gupta 1
- Bin Hu 1
- David Randolph IV 1
- Jaehwan Jeong 1
- Michael Liu 1
- Maria Mazzone 1
- Mohammad Mahmoudi Meymand 1
- Lucy Ogaz 1
- Jayeon Park 1
- Tahseen Rabbani 1
- Paula Rodriguez 1
- Pavit Singh 1
- Rakshith Sharma Srinivasa 1
- Utkarsh Tyagi 1
- Paula Vergara 1
- Zihao Wang 1