GuideWeb: A Benchmark for Automatic In-App Guide Generation on Real-World Web UIs

Chengguang Gan, Yunhao Liang, Yoshihiro Tsujii, Tatsunori Mori, Shiwen Ni, Hiroki Itoh


Abstract
Digital Adoption Platforms (DAPs) provide web-based overlays that deliver operational guidance and contextual hints to help users navigate complex websites. Although modern DAP tools enable non-experts to author such guidance, maintaining guides remains labor-intensive because website layouts and functionalities evolve continuously. In this work, we introduce GuideWeb, a human-verified benchmark for automatic in-app guide proposal on real-world web UIs. GuideWeb formulates the task as selecting guide target elements grounded in the webpage and generating concise guide text aligned with user intent. We make the target audience and guide-worthiness criteria explicit, and construct annotations through LLM-assisted candidate discovery followed by human verification, including deletion, rewriting, grounding correction, and addition of missed targets. We also propose an evaluation suite that jointly measures guide target selection, DOM grounding, and generated intent/guide-text quality. Experiments show that our proposed GuideWeb Agent achieves 30.79% F1 in guide target prediction, while obtaining BLEU scores of 44.94 for intent generation and 21.34 for guide-text generation. Existing baselines perform substantially worse, highlighting that reliable automatic guide proposal remains challenging on real-world web UIs.
Anthology ID:
2026.findings-aacl.77
Volume:
Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026
Month:
November
Year:
2026
Address:
Hengqin, China
Editors:
Yue Zhang, Ruifeng Xu, Ayu Purwarianti, Tong Rong, Jun Suzuki
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1519–1535
Language:
URL:
https://aclanthology.org/2026.findings-aacl.77/
DOI:
Bibkey:
Cite (ACL):
Chengguang Gan, Yunhao Liang, Yoshihiro Tsujii, Tatsunori Mori, Shiwen Ni, and Hiroki Itoh. 2026. GuideWeb: A Benchmark for Automatic In-App Guide Generation on Real-World Web UIs. In Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026, pages 1519–1535, Hengqin, China. Association for Computational Linguistics.
Cite (Informal):
GuideWeb: A Benchmark for Automatic In-App Guide Generation on Real-World Web UIs (Gan et al., Findings 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.findings-aacl.77.pdf