GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalization

Zhiyuan Zhang, Ruixuan Luo, Qi Su, Xu Sun


Abstract
Recently, Sharpness-Aware Minimization (SAM) algorithm has shown state-of-the-art generalization abilities in vision tasks. It demonstrates that flat minima tend to imply better generalization abilities. However, it has some difficulty implying SAM to some natural language tasks, especially to models with drastic gradient changes, such as RNNs. In this work, we analyze the relation between the flatness of the local minimum and its generalization ability from a novel and straightforward theoretical perspective. We propose that the shift of the training and test distributions can be equivalently seen as a virtual parameter corruption or perturbation, which can explain why flat minima that are robust against parameter corruptions or perturbations have better generalization performances. On its basis, we propose a Gradient-Strength based Adaptive Sharpness-Aware Minimization (GA-SAM) algorithm to help to learn algorithms find flat minima that generalize better. Results in various language benchmarks validate the effectiveness of the proposed GA-SAM algorithm on natural language tasks.
Anthology ID:
2022.emnlp-main.257
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Editors:
Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3888–3903
Language:
URL:
https://aclanthology.org/2022.emnlp-main.257
DOI:
10.18653/v1/2022.emnlp-main.257
Bibkey:
Cite (ACL):
Zhiyuan Zhang, Ruixuan Luo, Qi Su, and Xu Sun. 2022. GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3888–3903, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalization (Zhang et al., EMNLP 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.emnlp-main.257.pdf