2026

Post-training pruning compresses Large Language Models (LLMs) without full-model retraining, but performance degrades sharply at high sparsity. We identify two limitations in existing methods: local importance estimates miss global spectral structure, and uniform within-block sparsity ignores heterogeneous sensitivity across transformer projections. We propose PRISM, a post-training pruning framework with training-free mask construction and optional fixed-mask reconstruction optimization. PRISM augments activation-based scoring with two adaptive-rank SVD-derived signals for principal-subspace and residual-aware importance, combines the factors through parameter-free CV-based fusion, and allocates sparsity non-uniformly across attention and MLP projections using a gradient-weighted matrix-level sensitivity proxy. Across six Qwen3 and LLaMA models, PRISM consistently outperforms existing pruning baselines, yielding up to 16.8% relative improvement in WikiText-2 perplexity under the evaluated sparsity settings.
On-policy Direct Preference Optimization is commonly evaluated using held-out pairwise accuracy and LLM-judge win rate, but these metrics can reward different behaviors. In controlled Qwen3-4B ablations, interventions that improve a log-prob preference probe do not necessarily improve judged generations, and interventions that win generation judging need not improve the probe. The clearest gains in judge win rate come from direction correction, which strongly improves generation evaluation while leaving pairwise accuracy nearly unchanged. Output-level analysis shows that this advantage is driven primarily by response organization: the judged-winning model wins formatting on 40/50 prompts but wins instruction-following on only 11/50. Stripping markdown does not remove the advantage, suggesting that the effect is not merely markdown decoration but broader response organization. These results suggest that in this setting, on-policy DPO can optimize judge-preferred response organization without corresponding gains on a held-out preference probe.