Xiaojun Bi
Author directory2026
Are We There Yet? Assessing Computer-Use Agents for Blind Users’ Accessible Interaction with Desktop Applications
Satwik Ram Kodandaram | Monalika Padma Reddy | Xiaojun Bi | Jiawei Zhou | I. Ramakrishnan | Vikas Ashok
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Satwik Ram Kodandaram | Monalika Padma Reddy | Xiaojun Bi | Jiawei Zhou | I. Ramakrishnan | Vikas Ashok
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Computer-use agents are emerging as a paradigm for agentic human-AI interaction, combining language reasoning with multimodal interface grounding to operate GUIs. Yet their effectiveness for blind screen-reader users in real-world desktop workflows remains unclear. We present a three-week diary study with 8 blind users using OLLA, a screen-reader accessible CUA prototype, collecting 1,258commands across 12 applications with screenshots, UI trees, model responses, and action traces. We evaluate GPT-5 during deployment and re-execute the same commands with four additional models. GPT-5 achieved the highest success rate at 52.5%. Trace analysis reveals grounding, planning, constraint-tracking, and termination failures, while interviews reveal beyond-automation needs.
2025
DongbaMIE: A Multimodal Information Extraction Dataset for Evaluating Semantic Understanding of Dongba Pictograms
Xiaojun Bi | Shuo Li | Junyao Xing | Ziyue Wang | Fuwen Luo | Weizheng Qiao | Lu Han | Ziwei Sun | Peng Li | Yang Liu
Findings of the Association for Computational Linguistics: EMNLP 2025
Xiaojun Bi | Shuo Li | Junyao Xing | Ziyue Wang | Fuwen Luo | Weizheng Qiao | Lu Han | Ziwei Sun | Peng Li | Yang Liu
Findings of the Association for Computational Linguistics: EMNLP 2025
Dongba pictographic is the only pictographic script still in use in the world. Its pictorial ideographic features carry rich cultural and contextual information. However, due to the lack of relevant datasets, research on semantic understanding of Dongba hieroglyphs has progressed slowly. To this end, we constructed DongbaMIE - the first dataset focusing on multimodal information extraction of Dongba pictographs. The dataset consists of images of Dongba hieroglyphic characters and their corresponding semantic annotations in Chinese. It contains 23,530 sentence-level and 2,539 paragraph-level high-quality text-image pairs. The annotations cover four semantic dimensions: object, action, relation and attribute. Systematic evaluation of mainstream multimodal large language models shows that the models are difficult to perform information extraction of Dongba hieroglyphs efficiently under zero-shot and few-shot learning. Although supervised fine-tuning can improve the performance, accurate extraction of complex semantics is still a great challenge at present.