Ishitwa Viranchi

Author directory

2026

Autonomous GUI agents have achieved near-perfect task completion on touchscreen benchmarks by tapping directly at target coordinates. However, smart televisions, game consoles, and set-top boxes require users to navigate with a directional pad (D-PAD), turning element selection into a planning problem over an unknown focus-transition graph. We study autonomous task completion on streaming TV devices, where an agent navigates 68 streaming applications using only D-PAD inputs to perform four tasks: login, reaching subscription plans, content search, and settings navigation. We formalize this setting as a partially observable Markov decision process and compare two architectures using three open-weight vision-language models (Google Gemma 3 at 4B and 27B parameters, and Alibaba Qwen3-VL at 235B): (i) a VLM that maps screenshots directly to D-PAD actions, following the paradigm that dominates current benchmarks, and (ii) a system that maintains an explicit model of focus transition dynamics via Bayesian belief updates and graph-based path planning. The dynamics-aware system improves login success for every model tested, by a mean of 8.8 percentage points, and for the Gemma models its relative advantage widens on tasks requiring extended navigation. The benefit is inverse to model scale, ranging from 16.2 points for the 4B model to 2.9 points for the 235B model. A 4B model with dynamics awareness (32.4%) outperforms a 27B model without it (26.5%), indicating that transition knowledge can outweigh a 6.75×increase in model parameters.