Mrunal Shah
Author directory2026
Dynamics-Aware Navigation for Autonomous UI Agents on Directional-Input Devices
Ishitwa Viranchi | Akshintala Nagesh Kashyap | Mrunal Shah
Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
Ishitwa Viranchi | Akshintala Nagesh Kashyap | Mrunal Shah
Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
Autonomous GUI agents have achieved near-perfect task completion on touchscreen benchmarks by tapping directly at target coordinates. However, smart televisions, game consoles, and set-top boxes require users to navigate with a directional pad (D-PAD), turning element selection into a planning problem over an unknown focus-transition graph. We study autonomous task completion on streaming TV devices, where an agent navigates 68 streaming applications using only D-PAD inputs to perform four tasks: login, reaching subscription plans, content search, and settings navigation. We formalize this setting as a partially observable Markov decision process and compare two architectures using three open-weight vision-language models (Google Gemma 3 at 4B and 27B parameters, and Alibaba Qwen3-VL at 235B): (i) a VLM that maps screenshots directly to D-PAD actions, following the paradigm that dominates current benchmarks, and (ii) a system that maintains an explicit model of focus transition dynamics via Bayesian belief updates and graph-based path planning. The dynamics-aware system improves login success for every model tested, by a mean of 8.8 percentage points, and for the Gemma models its relative advantage widens on tasks requiring extended navigation. The benefit is inverse to model scale, ranging from 16.2 points for the 4B model to 2.9 points for the 235B model. A 4B model with dynamics awareness (32.4%) outperforms a 27B model without it (26.5%), indicating that transition knowledge can outweigh a 6.75×increase in model parameters.