<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="viranchi-etal-2026-dynamics">
    <titleInfo>
        <title>Dynamics-Aware Navigation for Autonomous UI Agents on Directional-Input Devices</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ishitwa</namePart>
        <namePart type="family">Viranchi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Akshintala</namePart>
        <namePart type="given">Nagesh</namePart>
        <namePart type="family">Kashyap</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mrunal</namePart>
        <namePart type="family">Shah</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Ehsan</namePart>
            <namePart type="family">Kamalloo</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nicolas</namePart>
            <namePart type="family">Gontier</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nikolai</namePart>
            <namePart type="family">Rozanov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Maryam</namePart>
            <namePart type="family">Hashemzadeh</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Akhil</namePart>
            <namePart type="family">Arora</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-485-9</identifier>
    </relatedItem>
    <abstract>Autonomous GUI agents have achieved near-perfect task completion on touchscreen benchmarks by tapping directly at target coordinates. However, smart televisions, game consoles, and set-top boxes require users to navigate with a directional pad (D-PAD), turning element selection into a planning problem over an unknown focus-transition graph. We study autonomous task completion on streaming TV devices, where an agent navigates 68 streaming applications using only D-PAD inputs to perform four tasks: login, reaching subscription plans, content search, and settings navigation. We formalize this setting as a partially observable Markov decision process and compare two architectures using three open-weight vision-language models (Google Gemma 3 at 4B and 27B parameters, and Alibaba Qwen3-VL at 235B): (i) a VLM that maps screenshots directly to D-PAD actions, following the paradigm that dominates current benchmarks, and (ii) a system that maintains an explicit model of focus transition dynamics via Bayesian belief updates and graph-based path planning. The dynamics-aware system improves login success for every model tested, by a mean of 8.8 percentage points, and for the Gemma models its relative advantage widens on tasks requiring extended navigation. The benefit is inverse to model scale, ranging from 16.2 points for the 4B model to 2.9 points for the 235B model. A 4B model with dynamics awareness (32.4%) outperforms a 27B model without it (26.5%), indicating that transition knowledge can outweigh a 6.75×increase in model parameters.</abstract>
    <identifier type="citekey">viranchi-etal-2026-dynamics</identifier>
    <location>
        <url>https://aclanthology.org/2026.realm-1.83/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>1205</start>
            <end>1212</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Dynamics-Aware Navigation for Autonomous UI Agents on Directional-Input Devices
%A Viranchi, Ishitwa
%A Kashyap, Akshintala Nagesh
%A Shah, Mrunal
%Y Kamalloo, Ehsan
%Y Gontier, Nicolas
%Y Rozanov, Nikolai
%Y Hashemzadeh, Maryam
%Y Arora, Akhil
%S Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-485-9
%F viranchi-etal-2026-dynamics
%X Autonomous GUI agents have achieved near-perfect task completion on touchscreen benchmarks by tapping directly at target coordinates. However, smart televisions, game consoles, and set-top boxes require users to navigate with a directional pad (D-PAD), turning element selection into a planning problem over an unknown focus-transition graph. We study autonomous task completion on streaming TV devices, where an agent navigates 68 streaming applications using only D-PAD inputs to perform four tasks: login, reaching subscription plans, content search, and settings navigation. We formalize this setting as a partially observable Markov decision process and compare two architectures using three open-weight vision-language models (Google Gemma 3 at 4B and 27B parameters, and Alibaba Qwen3-VL at 235B): (i) a VLM that maps screenshots directly to D-PAD actions, following the paradigm that dominates current benchmarks, and (ii) a system that maintains an explicit model of focus transition dynamics via Bayesian belief updates and graph-based path planning. The dynamics-aware system improves login success for every model tested, by a mean of 8.8 percentage points, and for the Gemma models its relative advantage widens on tasks requiring extended navigation. The benefit is inverse to model scale, ranging from 16.2 points for the 4B model to 2.9 points for the 235B model. A 4B model with dynamics awareness (32.4%) outperforms a 27B model without it (26.5%), indicating that transition knowledge can outweigh a 6.75×increase in model parameters.
%U https://aclanthology.org/2026.realm-1.83/
%P 1205-1212

Markdown (Informal)

[Dynamics-Aware Navigation for Autonomous UI Agents on Directional-Input Devices](https://aclanthology.org/2026.realm-1.83/) (Viranchi et al., REALM 2026)

• Dynamics-Aware Navigation for Autonomous UI Agents on Directional-Input Devices (Viranchi et al., REALM 2026)

ACL

• Ishitwa Viranchi, Akshintala Nagesh Kashyap, and Mrunal Shah. 2026. Dynamics-Aware Navigation for Autonomous UI Agents on Directional-Input Devices. In Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026), pages 1205–1212, Budapest, Hungary. Association for Computational Linguistics.