When applying LLM agents to automatic program repair (APR), existing systems often produce Draft Patches—partially correct fixes that either fail to fully resolve bugs or overfit public test cases due to incomplete contextual reasoning and limited test coverage. To address this, we present , a general context-aware refinement agent that evolves existing APR agents by transforming Draft Patches into robust and correct fixes. Operating in a black-box, plug-and-play manner, augments diverse APR systems without model retraining or architectural changes. tackles three challenges: (1) resolving ambiguous issue context through structured contextualization, (2) improving patch quality via test-time scaling and diversified generation, and (3) consolidating partial fixes through LLM-driven refinement. We integrate into multiple APR systems to demonstrate its generality and system-agnostic design. On SWE-Bench Lite, achieves state-of-the-art performance, improving AutoCodeRover by 14.67% to 51.67% resolution. %cOn SWE-Bench Verified, it improves resolution rates by 12.2%, while yielding an average 14% gain across diverse APR systems. These results identify refinement as a missing yet essential layer in APR pipelines and highlight agent-level refinement as a practical path toward more correct and robust program repair systems. We also open source our code : https://github.com/AnvithPabba/Refine-Program-Repair.

In the era of evaluating large language models (LLMs), data contamination has become an increasingly prominent concern. To address this risk, LLM benchmarking has evolved from a static to a dynamic paradigm. In this work, we conduct an in-depth analysis of existing static and dynamic benchmarks for evaluating LLMs. We first examine methods that enhance static benchmarks and identify their inherent limitations. We then highlight a critical gap—the lack of standardized criteria for evaluating dynamic benchmarks. Based on this observation, we propose a series of optimal design principles for dynamic benchmarking and analyze the limitations of existing dynamic benchmarks.This survey provides a concise yet comprehensive overview of recent advancements in data contamination research, offering valuable insights and a clear guide for future research efforts. We maintain a GitHub repository to continuously collect both static and dynamic benchmarking methods for LLMs. The repository can be found at this link.

Vision-Language Models (VLMs) have demonstrated great potential in real-world applications. While existing research primarily focuses on improving their accuracy, the efficiency remains underexplored. Given the real-time demands of many applications and the high inference overhead of VLMs, efficiency robustness is a critical issue. However, previous studies evaluate efficiency robustness under unrealistic assumptions, requiring access to the model architecture and parameters—an impractical scenario in ML-as-a-service settings, where VLMs are deployed via inference APIs. To address this gap, we propose VLMInferSlow, a novel approach for evaluating VLM efficiency robustness in a realistic black-box setting. VLMInferSlow incorporates fine-grained efficiency modeling tailored to VLM inference and leverages zero-order optimization to search for adversarial examples. Experimental results show that VLMInferSlow generates adversarial images with imperceptible perturbations, increasing the computational cost by up to 128.47%. We hope this research raises the community’s awareness about the efficiency robustness of VLMs.

Despite much success in natural language processing (NLP), pre-trained language models typically lead to a high computational cost during inference. Multi-exit is a mainstream approach to address this issue by making a trade-off between efficiency and accuracy, where the saving of computation comes from an early exit. However, whether such saving from early-exiting is robust remains unknown. Motivated by this, we first show that directly adapting existing adversarial attack approaches targeting model accuracy cannot significantly reduce inference efficiency. To this end, we propose a simple yet effective attacking framework, SAME, a novel slowdown attack framework on multi-exit models, which is specially tailored to reduce the efficiency of the multi-exit models. By leveraging the multi-exit models’ design characteristics, we utilize all internal predictions to guide the adversarial sample generation instead of merely considering the final prediction. Experiments on the GLUE benchmark show that SAME can effectively diminish the efficiency gain of various multi-exit models by 80% on average, convincingly validating its effectiveness and generalization ability.