Sheng Zhang

Author directory

Other people with similar names: Sheng Zhang, Sheng Zhang, Sheng Zhang, Sheng Zhang, Sheng Zhang

Unverified author pages with similar names: Sheng Zhang


2026

We present Trace2Ground, a large-scale web GUI grounding dataset and benchmark built by parsing HTML/DOM to obtain candidate interface elements and using large language models to simulate interaction traces and natural-language instructions. Existing datasets primarily target instructions solvable by matching the target element’s visible text, while real-world GUI tasks may require grounding under user scenarios, duplicate candidates, and relational descriptions. Trace2Ground targets these under-explored stress regimes through three curated splits: PragmaMind, where instructions describe user scenarios without naming UI elements; ContextTwin, where multiple elements share identical visible text and the correct target must be selected using interaction context; and RelateLoc, where the target is specified via spatial or structural relations to nearby anchors. We construct the dataset via an automated pipeline that extracts candidates from HTML/DOM structure, synthesizes traces and instructions with LLMs, and applies multi-view visual checks. Strong specialist GUI grounders underperform on these regimes despite reaching around 90% on ScreenSpot, exposing a gap not addressed by existing GUI training data. Standard supervised fine-tuning on Trace2Ground improves the base model’s performance on our curated regimes by 42.3% on average and yields zero-shot transfer to standard GUI benchmarks, including a 9.8% average improvement on the non-web ScreenSpot-Pro benchmark across six desktop domains.