Towards Better Semantic Understanding of Mobile Interfaces

Srinivas Sunkara, Maria Wang, Lijuan Liu, Gilles Baechler, Yu-Chung Hsiao, Jindong Chen, Abhanshu Sharma, James W. W. Stout


Abstract
Improving the accessibility and automation capabilities of mobile devices can have a significant positive impact on the daily lives of countless users. To stimulate research in this direction, we release a human-annotated dataset with approximately 500k unique annotations aimed at increasing the understanding of the functionality of UI elements. This dataset augments images and view hierarchies from RICO, a large dataset of mobile UIs, with annotations for icons based on their shapes and semantics, and associations between different elements and their corresponding text labels, resulting in a significant increase in the number of UI elements and the categories assigned to them. We also release models using image-only and multimodal inputs; we experiment with various architectures and study the benefits of using multimodal inputs on the new dataset. Our models demonstrate strong performance on an evaluation set of unseen apps, indicating their generalizability to newer screens. These models, combined with the new dataset, can enable innovative functionalities like referring to UI elements by their labels, improved coverage and better semantics for icons etc., which would go a long way in making UIs more usable for everyone.
Anthology ID:
2022.coling-1.497
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
5636–5650
Language:
URL:
https://aclanthology.org/2022.coling-1.497
DOI:
Bibkey:
Cite (ACL):
Srinivas Sunkara, Maria Wang, Lijuan Liu, Gilles Baechler, Yu-Chung Hsiao, Jindong Chen, Abhanshu Sharma, and James W. W. Stout. 2022. Towards Better Semantic Understanding of Mobile Interfaces. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5636–5650, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Towards Better Semantic Understanding of Mobile Interfaces (Sunkara et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.497.pdf
Data
ImageNet