OoMMix: Out-of-manifold Regularization in Contextual Embedding Space for Text Classification

Seonghyeon Lee, Dongha Lee, Hwanjo Yu


Abstract
Recent studies on neural networks with pre-trained weights (i.e., BERT) have mainly focused on a low-dimensional subspace, where the embedding vectors computed from input words (or their contexts) are located. In this work, we propose a new approach, called OoMMix, to finding and regularizing the remainder of the space, referred to as out-of-manifold, which cannot be accessed through the words. Specifically, we synthesize the out-of-manifold embeddings based on two embeddings obtained from actually-observed words, to utilize them for fine-tuning the network. A discriminator is trained to detect whether an input embedding is located inside the manifold or not, and simultaneously, a generator is optimized to produce new embeddings that can be easily identified as out-of-manifold by the discriminator. These two modules successfully collaborate in a unified and end-to-end manner for regularizing the out-of-manifold. Our extensive evaluation on various text classification benchmarks demonstrates the effectiveness of our approach, as well as its good compatibility with existing data augmentation techniques which aim to enhance the manifold.
Anthology ID:
2021.acl-long.49
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Month:
August
Year:
2021
Address:
Online
Editors:
Chengqing Zong, Fei Xia, Wenjie Li, Roberto Navigli
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
590–599
Language:
URL:
https://aclanthology.org/2021.acl-long.49
DOI:
10.18653/v1/2021.acl-long.49
Bibkey:
Cite (ACL):
Seonghyeon Lee, Dongha Lee, and Hwanjo Yu. 2021. OoMMix: Out-of-manifold Regularization in Contextual Embedding Space for Text Classification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 590–599, Online. Association for Computational Linguistics.
Cite (Informal):
OoMMix: Out-of-manifold Regularization in Contextual Embedding Space for Text Classification (Lee et al., ACL-IJCNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.acl-long.49.pdf
Optional supplementary material:
 2021.acl-long.49.OptionalSupplementaryMaterial.zip
Video:
 https://aclanthology.org/2021.acl-long.49.mp4
Data
AG News