We present Trace2Ground, a large-scale web GUI grounding dataset and benchmark built by parsing HTML/DOM to obtain candidate interface elements and using large language models to simulate interaction traces and natural-language instructions. Existing datasets primarily target instructions solvable by matching the target element’s visible text, while real-world GUI tasks may require grounding under user scenarios, duplicate candidates, and relational descriptions. Trace2Ground targets these under-explored stress regimes through three curated splits: PragmaMind, where instructions describe user scenarios without naming UI elements; ContextTwin, where multiple elements share identical visible text and the correct target must be selected using interaction context; and RelateLoc, where the target is specified via spatial or structural relations to nearby anchors. We construct the dataset via an automated pipeline that extracts candidates from HTML/DOM structure, synthesizes traces and instructions with LLMs, and applies multi-view visual checks. Strong specialist GUI grounders underperform on these regimes despite reaching around 90% on ScreenSpot, exposing a gap not addressed by existing GUI training data. Standard supervised fine-tuning on Trace2Ground improves the base model’s performance on our curated regimes by 42.3% on average and yields zero-shot transfer to standard GUI benchmarks, including a 9.8% average improvement on the non-web ScreenSpot-Pro benchmark across six desktop domains.

Evolutionary prompt optimizers such as GEPA suffer from prompt bloat: each iteration appends rules and caveats, producing prompts that are 2–3× longer yet no more accurate. We trace this to three deficiencies - incomplete error observation, limited search diversity, and unreliable selection - and propose ESPO (Error-Structured Prompt Optimization), which decomposes prompt optimization into three phases: Diagnose clusters all training errors into structural patterns in one round; Propose generates candidates via four complementary strategies with independent biases; Select applies bootstrap stability selection. On seven public NLP benchmarks - Tweet, MMLU, GSM8K, HotpotQA, ScoNe, HoVer, and PUPA - ESPO wins 6/7 datasets with +3.76% average accuracy over the state-of-the-art (74.67% vs. 70.91% for GEPA), while producing prompts 47% shorter (1,004 vs. 1,878 chars) and faster at inference. Cross-model experiments across four additional student models (Gemma 3 12B, Mistral 14B, Qwen3 32B, Claude Haiku 4.5) show ESPO yields the best average accuracy on every model tested, with the largest gap on Qwen3 GSM8K (15.00% → 91.40%). A generalization bound (Appendix) grounds each phase in a corresponding term of the test-time gap, and the ablation confirms a key prediction: adding diversity without bootstrap selection actually hurts performance (-1.20%).

Visual agent models for automating human activities on Graphical User Interfaces (GUIs) have emerged as a promising research direction, driven by advances in large Vision Language Models (VLMs). A critical challenge in GUI automation is the precise grounding of interface elements across diverse platforms. Existing vision-only GUI agents directly ground elements from large and cluttered screenshots, requiring them to process substantial irrelevant information that compromises their accuracy. In addition, these approaches typically employ basic cross-entropy loss for learning grounding objectives, which fails to effectively capture grounding quality compared to established object detection metrics like Intersection-over-Union (IoU). To address these issues, we introduce R-VLM, a novel GUI grounding approach that leverages zoomed-in region proposals for precise element localization. We also propose an IoU-aware objective function that facilitates model convergence toward high IoU predictions. Our approach bridges the gap between VLMs and conventional object detection techniques, improving the state-of-the-art grounding accuracy by 13% across diverse GUI platforms on the GUI grounding benchmarks ScreenSpot and AgentStudio. In addition, our R-VLM approach shows 3.2-9.7% absolute accuracy improvements in GUI navigation tasks on the AITW and Mind2Web benchmarks.