Zhongnan Qu
Author directory2026
Trace2Ground: Scenario, Disambiguation, and Relation Traces for Web GUI Grounding
Ruozhen He | Zhongnan Qu | Sheng Zhang | Mesut Erhan Unal | Zhuowen Tu | Srikar Appalaraju | Jianbo Yuan | Shabnam Ghadar | Haofu Liao
Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
Ruozhen He | Zhongnan Qu | Sheng Zhang | Mesut Erhan Unal | Zhuowen Tu | Srikar Appalaraju | Jianbo Yuan | Shabnam Ghadar | Haofu Liao
Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
We present Trace2Ground, a large-scale web GUI grounding dataset and benchmark built by parsing HTML/DOM to obtain candidate interface elements and using large language models to simulate interaction traces and natural-language instructions. Existing datasets primarily target instructions solvable by matching the target element’s visible text, while real-world GUI tasks may require grounding under user scenarios, duplicate candidates, and relational descriptions. Trace2Ground targets these under-explored stress regimes through three curated splits: PragmaMind, where instructions describe user scenarios without naming UI elements; ContextTwin, where multiple elements share identical visible text and the correct target must be selected using interaction context; and RelateLoc, where the target is specified via spatial or structural relations to nearby anchors. We construct the dataset via an automated pipeline that extracts candidates from HTML/DOM structure, synthesizes traces and instructions with LLMs, and applies multi-view visual checks. Strong specialist GUI grounders underperform on these regimes despite reaching around 90% on ScreenSpot, exposing a gap not addressed by existing GUI training data. Standard supervised fine-tuning on Trace2Ground improves the base model’s performance on our curated regimes by 42.3% on average and yields zero-shot transfer to standard GUI benchmarks, including a 9.8% average improvement on the non-web ScreenSpot-Pro benchmark across six desktop domains.