Phuong T. Nguyen
Author directory2026
XRepoTest: Benchmarking Multilingual Repository-Level Unit Test Generation for Large Language Models
Dung Le Quang | Dong Cao Van | Nam Le Hai | Linh Ngo Van | Thi-Mai-Anh Bui | Phuong T. Nguyen
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Dung Le Quang | Dong Cao Van | Nam Le Hai | Linh Ngo Van | Thi-Mai-Anh Bui | Phuong T. Nguyen
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Large language models (LLMs) have shown promise for automated unit test generation, but existing evaluations largely rely on standalone settings and a narrow set of programming languages, overestimating real-world readiness. We introduce XRepoTest, a multilingual, repository-level benchmark for unit test generation spanning five underexplored languages: Rust, Go, Julia, PHP, and Ruby. XRepoTest evaluates tests under realistic repository constraints using a containerized execution framework and multiple context augmentation strategies, including file-level, LSP-based, and retrieval-based context. Beyond standard metrics such as test pass rate and coverage, we propose Invocation Rate (IR) to assess whether generated tests meaningfully exercise the intended functionality. Experiments with 14 state-of-the-art LLMs (e.g., Claude 4.5, GPT-5.2, DeepSeek V4-pro, and Qwen families) reveal a substantial gap between standalone and repository-level performance, as well as trade-offs between richer context and test reliability. Overall, XRepoTest provides a challenging and informative benchmark to advance scalable, robust, and multilingual unit test generation in realistic software environments.