We present Trace2Ground, a large-scale web GUI grounding dataset and benchmark built by parsing HTML/DOM to obtain candidate interface elements and using large language models to simulate interaction traces and natural-language instructions. Existing datasets primarily target instructions solvable by matching the target element’s visible text, while real-world GUI tasks may require grounding under user scenarios, duplicate candidates, and relational descriptions. Trace2Ground targets these under-explored stress regimes through three curated splits: PragmaMind, where instructions describe user scenarios without naming UI elements; ContextTwin, where multiple elements share identical visible text and the correct target must be selected using interaction context; and RelateLoc, where the target is specified via spatial or structural relations to nearby anchors. We construct the dataset via an automated pipeline that extracts candidates from HTML/DOM structure, synthesizes traces and instructions with LLMs, and applies multi-view visual checks. Strong specialist GUI grounders underperform on these regimes despite reaching around 90% on ScreenSpot, exposing a gap not addressed by existing GUI training data. Standard supervised fine-tuning on Trace2Ground improves the base model’s performance on our curated regimes by 42.3% on average and yields zero-shot transfer to standard GUI benchmarks, including a 9.8% average improvement on the non-web ScreenSpot-Pro benchmark across six desktop domains.

We introduce DavIR, a model-based data selection method for post-training Large Language Models. DavIR generalizes Reducible Holdout Loss to core-set selection problem of causal language modeling, and quantifies the learnability of a given datum with respect to a pre-trained LLM based on relative reduction in loss during fine-tuning, a metric we show to be closely related to the implicit reward model described in Direct Preference Optimization (DPO). We show that 6% of Alpaca dataset selected with DavIR can steer both the LLaMA and Gemma model family to produce superior performance compared to the same models trained on the full 52K dataset. We also show that Alpaca dataset compressed with DavIR can be combined with GSM8K dataset to effectively balance open-domain freeform QA and mathematical reasoning capabilities. Finally, we apply the DavIR objective to DPO and develop a normalized DavIR-DPO objective which improves alignment performance of Zephyr-7B-SFT model by 8% (relative) on AlpacaEval, compared against training on vanilla DPO objective.

In this work, we present InfiMM, an advanced Multimodal Large Language Model that adapts to intricate vision-language tasks. InfiMM, inspired by the Flamingo architecture, distinguishes itself through the utilization of large-scale training data, comprehensive training strategies, and diverse large language models. This approach ensures the preservation of Flamingo’s foundational strengths while simultaneously introducing augmented capabilities. Empirical evaluations across a variety of benchmarks underscore InfiMM’s remarkable capability in multimodal understanding. The code can be found at: https://anonymous.4open.science/r/infimm-zephyr-F60C/.

We present Expert-Token-Routing, a unified generalist framework that facilitates seamless integration of multiple expert LLMs. Our framework represents expert LLMs as special expert tokens within the vocabulary of a meta LLM. The meta LLM can route to an expert LLM like generating new tokens. Expert-Token-Routing not only supports learning the implicit expertise of expert LLMs from existing instruction dataset but also allows for dynamic extension of new expert LLMs in a plug-and-play manner. It also conceals the detailed collaboration process from the user’s perspective, facilitating interaction as though it were a singular LLM. Our framework outperforms various existing multi-LLM collaboration paradigms across benchmarks that incorporate six diverse expert domains, demonstrating effectiveness and robustness in building generalist LLM system via synergizing multiple expert LLMs.