Wanyue Zhang
Author directory2026
SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios
Juntao Cheng | Wanyue Zhang | Zhiwei Yu | Shuo Ren | Zheqi He | Shaoxuan Xie | Guocai Yao | Jieru Lin | Börje F. Karlsson | Jiajun Zhang
Findings of the Association for Computational Linguistics: EMNLP 2026
Juntao Cheng | Wanyue Zhang | Zhiwei Yu | Shuo Ren | Zheqi He | Shaoxuan Xie | Guocai Yao | Jieru Lin | Börje F. Karlsson | Jiajun Zhang
Findings of the Association for Computational Linguistics: EMNLP 2026
Tangible control interfaces (TCIs), such as appliance panels, remotes, elevators, and embedded GUIs, are a fundamental component of everyday human-built environments. Interacting with these interfaces requires agents not only to ground language in visual observations, but also to execute actions, track temporally evolving state changes, and verify whether intended outcomes have been achieved. However, existing benchmarks predominantly evaluate open-loop perception or single-step action execution, failing to capture this continuous cycle of interaction, feedback, and correction. We introduce SWITCH, a benchmark for closed-loop interactive reasoning with TCIs inrealistic egocentric environments1. SWITCH comprises 1,170 temporally interactive videos across diverse functional categories, providing structured annotations of instructions, actions,state transitions, outcomes, and recovery behaviors over time. To probe generative world modeling, SWITCH also evaluates video generation models on interaction-centered tasks using both LLM-as-judge and human evaluation.Experiments with frontier proprietary and opensource multimodal models reveal persistent weaknesses in fine-grained visual-temporal perception, outcome verification, and error recovery, highlighting SWITCH as a testbed for closed-loop embodied intelligence