Qiaojuan Peng
Author directory2026
Beyond Gradient Egalitarianism: Dual-Adaptive C-AUM Framework for Few-Shot Fine-Tuning of Large Language Models
Qiaojuan Peng | Xiong Luo | Yuqi Yuan | Wenbing Zhao
Findings of the Association for Computational Linguistics: EMNLP 2026
Qiaojuan Peng | Xiong Luo | Yuqi Yuan | Wenbing Zhao
Findings of the Association for Computational Linguistics: EMNLP 2026
While large language models (LLMs) demonstrate strong text classification capabilities, adapting them to specialized, few-shot industrial domains remains challenging. Vanilla supervised fine-tuning (SFT) applies uniform weighting to training samples and may overfit easy instances while insufficiently optimizing ambiguous boundary samples. Static regularization may further limit adaptation by applying the same constraint throughout training. To address these limitations, we propose cumulative area under the margin (C-AUM), a fine-tuning framework tailored to autoregressive LLM classification. C-AUM tracks the cumulative margin of the final decision token as an instance-level difficulty signal and uses it to coordinate AUM-aware instance weighting with difficulty-aware dynamic LoRA regularization. We also construct and release an expert-verified few-shot text classification benchmark for the steel industry, comprising 1,070 samples across seven fine-grained categories. Evaluations across six representative LLMs show that C-AUM achieves comparable or improved performance over vanilla SFT, with a maximum macro F1 gain of 1.99 percentage points. Our dataset and code are publicly available at https://github.com/pengdada98/C-AUM.