Recently, the Muon optimizer has demonstrated strong efficiency in large model training through matrix orthogonalization. At each optimization step, Muon orthogonalizes the raw update matrix via Newton-Schulz (NS) iterations before applying it to model parameters. However, orthogonalization requires repeated matrix multiplications whose computational cost scales with O(n³) for an n× n matrix, making the overhead prohibitive at large model scales. In this paper, we observe that the geometric angle derived from cosine similarity between consecutive orthogonal update matrices remains highly stable during training, a phenomenon that we partially explain theoretically and support empirically. Based on this angular stability, we propose LazyMuon, which skips computationally expensive NS iterations through intermittent linear interpolation and employs a residual correction mechanism to mitigate accumulated approximation errors. This design ensures fast iteration while preserving Muon’s orthogonal manifold constraints. Empirical evaluations on language and vision tasks demonstrate that LazyMuon achieves speedups of up to 1.34× and 1.31× over Muon, respectively, with negligible degradation in convergence.

Reinforcement Learning with Verifiable Rewards (RLVR) for LLM reasoning is often framed as balancing exploration and exploitation in action space, typically operationalized with token-level proxies (e.g., output entropy or confidence). We argue that this apparent trade-off is largely a measurement artifact: token-level statistics reflect next-token uncertainty rather than how reasoning progresses over multi-token semantic structures. We therefore study exploration and exploitation in the hidden-state space of response trajectories. We use Effective Rank (ER) to quantify representational exploration and introduce its temporal derivatives, Effective Rank Velocity (ERV) and Effective Rank Acceleration (ERA), to characterize exploitative refinement dynamics. Empirically and theoretically, ER and ERV exhibit near-zero correlation in semantic space, suggesting the two capacities can be improved simultaneously. Motivated by this, we propose Velocity-Exploiting Rank Learning (VERL), which shapes the RL advantage with an auxiliary signal derived from ER/ERV and uses the more stable ERA as a meta-control variable to adaptively balance the incentives. Across multiple base models, RL algorithms, and reasoning benchmarks, VERL yields consistent improvements, including large gains on challenging tasks (e.g., 21.4% in Gaokao 2024).

Text-driven indoor scene generation and editing require an intermediate representation that language models can both produce and revise. Existing LLM-based systems often rely on scene graphs or global constraint lists, which are compact but underspecify local geometry and make instruction-based edits difficult to localize. We frame the task as structured program generation and local program repair, and propose the Hierarchical Descriptive Scene Language (HDSL), an XML/CSS-style domain-specific language in which rooms, regions, objects, and support surfaces form a tree with parent-relative local coordinates. Our pipeline generates HDSL subtrees recursively with bounded verification, grounds non-virtual nodes through multimodal asset retrieval, and applies force-directed layout optimization to repair boundary and collision errors. For editing, Hierarchical Retrieval-Augmented Generation retrieves the relevant subtree, rewrites only that local context, and merges the result back through a deterministic three-way merge. In our reproduced benchmark, HDSL improves average object coverage, text-scene alignment, and generation time over full text-to-scene baselines while remaining competitive with layout-only reproductions on geometry metrics. On thirty paired editing instructions, HRAG reduces token use by 67.6% and runtime by 68.1%, and cuts non-target removals by 87.5% over full-scene rewriting; attribute ablations confirm that local coordinates and virtual/leaf flags are necessary for valid, locality-preserving edits.

The emergence of large language models (LLMs) has revolutionized AI development, yet their resource demands beyond a single cluster or even datacenter, limiting accessibility to well-resourced organizations. Decentralized training has emerged as a promising paradigm to leverage dispersed resources across clusters, datacenters and even regions, offering the potential to democratize LLM development for broader communities. As the first comprehensive exploration of this emerging field, we present decentralized LLM training as a resource-driven paradigm and categorize existing efforts into community-driven and organizational approaches. We further clarify this through: (1) a comparison with related paradigms, (2) characterization of decentralized resources, and (3) a taxonomy of recent advancements. We also provide up-to-date case studies and outline future directions to advance research in decentralized LLM training.

In recent years, researchers tend to pre-train ever-larger language models to explore the upper limit of deep models. However, large language model pre-training costs intensive computational resources, and most of the models are trained from scratch without reusing the existing pre-trained models, which is wasteful. In this paper, we propose bert2BERT, which can effectively transfer the knowledge of an existing smaller pre-trained model to a large model through parameter initialization and significantly improve the pre-training efficiency of the large model. Specifically, we extend the previous function-preserving method proposed in computer vision on the Transformer-based language model, and further improve it by proposing a novel method, advanced knowledge for large model’s initialization. In addition, a two-stage learning method is proposed to further accelerate the pre-training. We conduct extensive experiments on representative PLMs (e.g., BERT and GPT) and demonstrate that (1) our method can save a significant amount of training cost compared with baselines including learning from scratch, StackBERT and MSLT; (2) our method is generic and applicable to different types of pre-trained models. In particular, bert2BERT saves about 45% and 47% computational cost of pre-training BERT _BASE and GPT _BASE by reusing the models of almost their half sizes.