Recent advances in large language models (LLMs) have enabled software engineering agents to solve complex code modification tasks. However, most existing approaches rely on execution feedback from Docker-based environments, which require dependency-complete setup, program execution, and test validation, making agent training costly and difficult to scale. We propose SWE-World, a Docker-free framework that replaces containerized execution during downstream agent optimization with a learned surrogate environment. Trained on real agent-environment interaction traces, SWE-World predicts both step-level execution feedback and final test outcomes, preserving the standard agent-environment loop without repeatedly constructing or running Docker environments. This surrogate also enables test-time scaling (TTS) by scoring multiple candidate trajectories through simulated evaluation. Experiments on SWE-bench Verified show that SWE-World improves Qwen2.5-Coder-32B-Instruct from 6.2% to 52.0% with Docker-free SFT, 55.0% with Docker-free RL, and 68.2% with TTS. The code is available at https://anonymous.4open.science/r/SWE-World-2D1E.

Code reasoning is essential for building AI systems that can understand, generate, and verify complex software. Yet most open-source code corpora focus on isolated snippets or single functions, failing to capture the long-range dependencies and cross-file interactions that define real-world repositories. We introduce MegaCode, a 300B repository-level corpus designed to support structured code reasoning and long-context modeling. MegaCode preserves full project structure—including build scripts, configuration files, and inter-file links—and is constructed through a multi-stage curation pipeline combining deduplication, rule-based filtering, and model-based quality assessment. Models pretrained on MegaCode demonstrate consistent gains on code reasoning and long-context benchmarks compared with those trained on flattened code datasets. Also, we validate the effectiveness of adding repo relevent infomation such as LSP, repo structure and repo summary. These results highlight the importance of maintaining repository structure for next-generation code intelligence. We release the dataset, curation pipeline, and evaluation suite to facilitate research on large-scale code pretraining and repository-level understanding.

Large language models (LLMs) demonstrate remarkable multilingual capabilities without being pre-trained on specially curated multilingual parallel corpora.It remains a challenging problem to explain the underlying mechanisms by which LLMs process multilingual texts.In this paper, we delve into the composition of Transformer architectures in LLMs to pinpoint language-specific regions.Specially, we propose a novel detection method, language activation probability entropy (LAPE), to identify language-specific neurons within LLMs.Based on LAPE, we conduct comprehensive experiments on several representative LLMs, such as LLaMA-2, BLOOM, and Mistral. Our findings indicate that LLMs’ proficiency in processing a particular language is predominantly due to a small subset of neurons, primarily situated in the models’ top and bottom layers.Furthermore, we showcase the feasibility to “steer” the output language of LLMs by selectively activating or deactivating language-specific neurons. Our research provides important evidence to the understanding and exploration of the multilingual capabilities of LLMs.

To facilitate the research on large language models (LLMs), this paper presents a comprehensive and unified library, LLMBox, to ease the development, use, and evaluation of LLMs. This library is featured with three main merits: (1) a unified data interface that supports the flexible implementation of various training strategies, (2) a comprehensive evaluation that covers extensive tasks, datasets, and models, and (3) more practical consideration, especially on user-friendliness and efficiency. With our library, users can easily reproduce existing methods, train new models, and conduct comprehensive performance comparisons. To rigorously test LLMBox, we conduct extensive experiments in a diverse coverage of evaluation settings, and experimental results demonstrate the effectiveness and efficiency of our library in supporting various implementations related to LLMs. The detailed introduction and usage guidance can be found at https://github.com/RUCAIBox/LLMBox.