Keyu Wen
Author directory2026
PRISM: Post-Training LLM Pruning via Residual Importance Signal Merging and Intra-Block Sparsity Allocation
Keyu Wen | Jianing Zhu | Kaiyang HU | Lijun Yang | Rui Lin
Findings of the Association for Computational Linguistics: EMNLP 2026
Keyu Wen | Jianing Zhu | Kaiyang HU | Lijun Yang | Rui Lin
Findings of the Association for Computational Linguistics: EMNLP 2026
Post-training pruning compresses Large Language Models (LLMs) without full-model retraining, but performance degrades sharply at high sparsity. We identify two limitations in existing methods: local importance estimates miss global spectral structure, and uniform within-block sparsity ignores heterogeneous sensitivity across transformer projections. We propose PRISM, a post-training pruning framework with training-free mask construction and optional fixed-mask reconstruction optimization. PRISM augments activation-based scoring with two adaptive-rank SVD-derived signals for principal-subspace and residual-aware importance, combines the factors through parameter-free CV-based fusion, and allocates sparsity non-uniformly across attention and MLP projections using a gradient-weighted matrix-level sensitivity proxy. Across six Qwen3 and LLaMA models, PRISM consistently outperforms existing pruning baselines, yielding up to 16.8% relative improvement in WikiText-2 perplexity under the evaluated sparsity settings.
Alignment or Formatting? Dissecting What On-Policy DPO Actually Learns
Keyu Wen | Yang Song | Lijun Yang | Rui Lin
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Keyu Wen | Yang Song | Lijun Yang | Rui Lin
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
On-policy Direct Preference Optimization is commonly evaluated using held-out pairwise accuracy and LLM-judge win rate, but these metrics can reward different behaviors. In controlled Qwen3-4B ablations, interventions that improve a log-prob preference probe do not necessarily improve judged generations, and interventions that win generation judging need not improve the probe. The clearest gains in judge win rate come from direction correction, which strongly improves generation evaluation while leaving pairwise accuracy nearly unchanged. Output-level analysis shows that this advantage is driven primarily by response organization: the judged-winning model wins formatting on 40/50 prompts but wins instruction-following on only 11/50. Stripping markdown does not remove the advantage, suggesting that the effect is not merely markdown decoration but broader response organization. These results suggest that in this setting, on-policy DPO can optimize judge-preferred response organization without corresponding gains on a held-out preference probe.