Are We There Yet? Assessing Computer-Use Agents for Blind Users’ Accessible Interaction with Desktop Applications

Satwik Ram Kodandaram, Monalika Padma Reddy, Xiaojun Bi, Jiawei Zhou, I. Ramakrishnan, Vikas Ashok


Abstract
Computer-use agents are emerging as a paradigm for agentic human-AI interaction, combining language reasoning with multimodal interface grounding to operate GUIs. Yet their effectiveness for blind screen-reader users in real-world desktop workflows remains unclear. We present a three-week diary study with 8 blind users using OLLA, a screen-reader accessible CUA prototype, collecting 1,258commands across 12 applications with screenshots, UI trees, model responses, and action traces. We evaluate GPT-5 during deployment and re-execute the same commands with four additional models. GPT-5 achieved the highest success rate at 52.5%. Trace analysis reveals grounding, planning, constraint-tracking, and termination failures, while interviews reveal beyond-automation needs.
Anthology ID:
2026.emnlp-main.1763
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
40189–40209
Language:
URL:
https://aclanthology.org/2026.emnlp-main.1763/
DOI:
Bibkey:
Cite (ACL):
Satwik Ram Kodandaram, Monalika Padma Reddy, Xiaojun Bi, Jiawei Zhou, I. Ramakrishnan, and Vikas Ashok. 2026. Are We There Yet? Assessing Computer-Use Agents for Blind Users’ Accessible Interaction with Desktop Applications. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 40189–40209, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Are We There Yet? Assessing Computer-Use Agents for Blind Users’ Accessible Interaction with Desktop Applications (Kodandaram et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.1763.pdf