Knowledge Editing (KE) aims to update specific facts in large language models (LLMs) without full retraining. Prior efforts sought to tune the knowledge layers of LLMs, achieving near-perfect performance in teacher-forced evaluations. However, under realistic autoregressive generation evaluation, their effectiveness remains limited by two core challenges: (1) even when methods successfully inject new knowledge, pre-trained capabilities can degrade, and (2) more critically, the injected parametric knowledge is not reliably expressed at inference time. To this end, we formalize knowledge editing as a two-stage problem and propose EtCon, an edit-then-consolidate paradigm that couples targeted edits with post-edit consolidation. Within EtCon, the edit stage applies Targeted Proximal Supervised Fine-Tuning (TPSFT) to update parametric knowledge with minimal disruption to pre-trained capabilities. The consolidation stage applies Group Relative Policy Optimization (GRPO) to align the model’s autoregressive generation with the edited parametric knowledge. Extensive experiments demonstrate that our instantiation (TPSFT + GRPO) improves editing reliability and real-world generalization while preserving pre-trained capabilities, and that EtCon consistently improves existing editing methods when instantiated with them.

We are currently in an era of fierce competition among various large language models (LLMs), continuously pushing the boundaries of benchmark performance. However, genuinely assessing the capabilities of these LLMs has become a challenging and critical issue due to potential data contamination. In this paper, we propose a novel and valuable method, Clean-Eval, which mitigates the issue of data contamination and evaluates the LLMs more cleanly. Clean-Eval employs a neural-based model to paraphrase and back-translate the contaminated data into a candidate set, generating expressions with the same meaning but in different surface forms. A semantic detector is then used to filter those generated low-quality samples to narrow down this candidate set. Candidates with moderate BLEURT scores against the original samples are selected as the final evaluation set. According to human assessment, this set is almost semantically equivalent to the original contamination set but expressed differently. We conduct experiments on 20 existing benchmarks across diverse tasks, and results demonstrate that Clean-Eval substantially restores the actual evaluation results on contaminated LLMs under both few-shot learning and fine-tuning scenarios.

The decoding algorithm is critical for open-ended text generation, transforming latent representations into coherent and meaningful outputs. This paper investigates the self-reinforcement effect in text generation and the effectiveness of a repetition penalty to mitigate it. However, determining the optimal repetition penalty value is challenging. To tackle this, we propose a forgetting mechanism that disregards distant tokens, reducing the burden of penalty selection. In addition, we introduce a length penalty to address overly short sentences caused by excessive penalties. Our penalty decoding approach incorporating three strategies helps resolve issues with sampling methods deviating from factual information. Experimental results demonstrate the efficacy of our approach in generating high-quality sentences resembling human output.