Emmanuel Helbert
Author directory2026
Learning New Facts with QLoRA: An Acquisition-Retention Frontier
Estelle Zheng | Sébastien Warichet | Emmanuel Helbert | Christophe Cerisara
Findings of the Association for Computational Linguistics: EMNLP 2026
Estelle Zheng | Sébastien Warichet | Emmanuel Helbert | Christophe Cerisara
Findings of the Association for Computational Linguistics: EMNLP 2026
Parameter-efficient fine-tuning (PEFT) is often assumed to preserve pretrained capabilities because it updates only a small number of parameters. We show that this assumption depends strongly on adapter capacity. We study factual acquisition in a controlled OpenStreetMap-derived benchmark where Qwen3-4B must learn anonymized geographic relations while retaining unrelated capabilities. Comparing FFT with QLoRA ranks 8, 16, 32, and 64, we find that rank induces a clear acquisition–retention frontier. Low-rank QLoRA preserves OOD performance but acquires fewer facts, whereas higher ranks improve same-fact paraphrase generalization at increasing cost to unrelated benchmarks. FFT behaves as a conservative baseline: it retains general capabilities well, but does not reach the highest factual-acquisition regime. Distributional, weight-space, and spectral diagnostics mirror this behavioral trade-off, with higher-rank QLoRA moving farther from the pretrained model. A separate math adaptation experiment shows a weaker frontier, suggesting that the effect is most pronounced when adaptation must install new factual associations rather than reinforce skills already supported by pretraining.