@inproceedings{guo-etal-2026-beyond-seeing,
title = "Beyond Seeing: Evaluating Multimodal {LLM}s on Tool-Enabled Image Perception, Transformation, and Reasoning",
author = "Guo, Xingang and
Tyagi, Utkarsh and
Gosai, Advait and
Vergara, Paula and
Park, Jayeon and
Montoya, Ernesto Gabriel Hern{\'a}ndez and
Zhang, Chen Bo Calvin and
Hu, Bin and
He, Yunzhong and
Liu, Bing and
Srinivasa, Rakshith Sharma",
editor = "Lee, Hung-Yi and
Dev, Sunipa and
Li, Jessy and
Elliott, Desmond",
booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.emnlp-main.1711/",
pages = "38981--39048",
ISBN = "979-8-89176-459-0",
abstract = "Think-with-images, where multimodal LLMs (MLLMs) actively manipulate visual inputs as part of their reasoning, has become a focal research direction for agentic multimodal systems, with a surge of prompting, supervised, and reinforcement learning (RL) methods teaching models to invoke vision tools such as cropping, zooming, and editing. Yet most established multimodal benchmarks use images whose critical content is already easy to perceive in a single pass, and many have saturated on frontier models. We introduce VisualToolBench (VTB), a benchmark deliberately curated so that critical visual content is dense, occluded, dispersed, rotated, or low-contrast, making active image manipulation a prerequisite for success. VTB contains 1,204 expert-authored, open-ended tasks (603 single-turn, 601 multi-turn) across five domains. Each task is paired with rubrics tagged by five competency categories (Visual Understanding, Truthfulness, Instruction Following, Reasoning, and Writing Style) that enable per-axis disentanglement. Across 26 evaluated MLLMs, even the strongest model reaches only 29{\%} accuracy, with Visual Understanding the consistent bottleneck and visual misinterpretation accounting for 32{\%} to 39{\%} of failures. Top models, moreover, converge to comparable accuracy through markedly different tool-use strategies, which we analyze in detail."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="guo-etal-2026-beyond-seeing">
<titleInfo>
<title>Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning</title>
</titleInfo>
<name type="personal">
<namePart type="given">Xingang</namePart>
<namePart type="family">Guo</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Utkarsh</namePart>
<namePart type="family">Tyagi</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Advait</namePart>
<namePart type="family">Gosai</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Paula</namePart>
<namePart type="family">Vergara</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jayeon</namePart>
<namePart type="family">Park</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Ernesto</namePart>
<namePart type="given">Gabriel</namePart>
<namePart type="given">Hernández</namePart>
<namePart type="family">Montoya</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Chen</namePart>
<namePart type="given">Bo</namePart>
<namePart type="given">Calvin</namePart>
<namePart type="family">Zhang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Bin</namePart>
<namePart type="family">Hu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yunzhong</namePart>
<namePart type="family">He</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Bing</namePart>
<namePart type="family">Liu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Rakshith</namePart>
<namePart type="given">Sharma</namePart>
<namePart type="family">Srinivasa</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-10</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
</titleInfo>
<name type="personal">
<namePart type="given">Hung-Yi</namePart>
<namePart type="family">Lee</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Sunipa</namePart>
<namePart type="family">Dev</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jessy</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Desmond</namePart>
<namePart type="family">Elliott</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Budapest, Hungary</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-459-0</identifier>
</relatedItem>
<abstract>Think-with-images, where multimodal LLMs (MLLMs) actively manipulate visual inputs as part of their reasoning, has become a focal research direction for agentic multimodal systems, with a surge of prompting, supervised, and reinforcement learning (RL) methods teaching models to invoke vision tools such as cropping, zooming, and editing. Yet most established multimodal benchmarks use images whose critical content is already easy to perceive in a single pass, and many have saturated on frontier models. We introduce VisualToolBench (VTB), a benchmark deliberately curated so that critical visual content is dense, occluded, dispersed, rotated, or low-contrast, making active image manipulation a prerequisite for success. VTB contains 1,204 expert-authored, open-ended tasks (603 single-turn, 601 multi-turn) across five domains. Each task is paired with rubrics tagged by five competency categories (Visual Understanding, Truthfulness, Instruction Following, Reasoning, and Writing Style) that enable per-axis disentanglement. Across 26 evaluated MLLMs, even the strongest model reaches only 29% accuracy, with Visual Understanding the consistent bottleneck and visual misinterpretation accounting for 32% to 39% of failures. Top models, moreover, converge to comparable accuracy through markedly different tool-use strategies, which we analyze in detail.</abstract>
<identifier type="citekey">guo-etal-2026-beyond-seeing</identifier>
<location>
<url>https://aclanthology.org/2026.emnlp-main.1711/</url>
</location>
<part>
<date>2026-10</date>
<extent unit="page">
<start>38981</start>
<end>39048</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings
%T Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning
%A Guo, Xingang
%A Tyagi, Utkarsh
%A Gosai, Advait
%A Vergara, Paula
%A Park, Jayeon
%A Montoya, Ernesto Gabriel Hernández
%A Zhang, Chen Bo Calvin
%A Hu, Bin
%A He, Yunzhong
%A Liu, Bing
%A Srinivasa, Rakshith Sharma
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F guo-etal-2026-beyond-seeing
%X Think-with-images, where multimodal LLMs (MLLMs) actively manipulate visual inputs as part of their reasoning, has become a focal research direction for agentic multimodal systems, with a surge of prompting, supervised, and reinforcement learning (RL) methods teaching models to invoke vision tools such as cropping, zooming, and editing. Yet most established multimodal benchmarks use images whose critical content is already easy to perceive in a single pass, and many have saturated on frontier models. We introduce VisualToolBench (VTB), a benchmark deliberately curated so that critical visual content is dense, occluded, dispersed, rotated, or low-contrast, making active image manipulation a prerequisite for success. VTB contains 1,204 expert-authored, open-ended tasks (603 single-turn, 601 multi-turn) across five domains. Each task is paired with rubrics tagged by five competency categories (Visual Understanding, Truthfulness, Instruction Following, Reasoning, and Writing Style) that enable per-axis disentanglement. Across 26 evaluated MLLMs, even the strongest model reaches only 29% accuracy, with Visual Understanding the consistent bottleneck and visual misinterpretation accounting for 32% to 39% of failures. Top models, moreover, converge to comparable accuracy through markedly different tool-use strategies, which we analyze in detail.
%U https://aclanthology.org/2026.emnlp-main.1711/
%P 38981-39048
Markdown (Informal)
[Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning](https://aclanthology.org/2026.emnlp-main.1711/) (Guo et al., EMNLP 2026)
ACL
- Xingang Guo, Utkarsh Tyagi, Advait Gosai, Paula Vergara, Jayeon Park, Ernesto Gabriel Hernández Montoya, Chen Bo Calvin Zhang, Bin Hu, Yunzhong He, Bing Liu, and Rakshith Sharma Srinivasa. 2026. Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 38981–39048, Budapest, Hungary. Association for Computational Linguistics.