Iterative Stratified Testing and Measurement for Automated Model Updates

Elizabeth Dekeyser, Nicholas Comment, Shermin Pei, Rajat Kumar, Shruti Rai, Fengtao Wu, Lisa Haverty, Kanna Shimizu


Abstract
Automating updates to machine learning systems is an important but understudied challenge in AutoML. The high model variance of many cutting-edge deep learning architectures means that retraining a model provides no guarantee of accurate inference on all sample types. To address this concern, we present Automated Data-Shape Stratified Model Updates (ADSMU), a novel framework that relies on iterative model building coupled with data-shape stratified model testing and improvement. Using ADSMU, we observed a 26% (relative) improvement in accuracy for new model use cases on a large-scale NLU system, compared to a naive (manually) retrained baseline and current cutting-edge methods.
Anthology ID:
2022.emnlp-industry.20
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track
Month:
December
Year:
2022
Address:
Abu Dhabi, UAE
Editors:
Yunyao Li, Angeliki Lazaridou
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
198–205
Language:
URL:
https://aclanthology.org/2022.emnlp-industry.20
DOI:
10.18653/v1/2022.emnlp-industry.20
Bibkey:
Cite (ACL):
Elizabeth Dekeyser, Nicholas Comment, Shermin Pei, Rajat Kumar, Shruti Rai, Fengtao Wu, Lisa Haverty, and Kanna Shimizu. 2022. Iterative Stratified Testing and Measurement for Automated Model Updates. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 198–205, Abu Dhabi, UAE. Association for Computational Linguistics.
Cite (Informal):
Iterative Stratified Testing and Measurement for Automated Model Updates (Dekeyser et al., EMNLP 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.emnlp-industry.20.pdf