Attentive Pooling with Learnable Norms for Text Representation

Chuhan Wu, Fangzhao Wu, Tao Qi, Xiaohui Cui, Yongfeng Huang


Abstract
Pooling is an important technique for learning text representations in many neural NLP models. In conventional pooling methods such as average, max and attentive pooling, text representations are weighted summations of the L1 or L∞ norm of input features. However, their pooling norms are always fixed and may not be optimal for learning accurate text representations in different tasks. In addition, in many popular pooling methods such as max and attentive pooling some features may be over-emphasized, while other useful ones are not fully exploited. In this paper, we propose an Attentive Pooling with Learnable Norms (APLN) approach for text representation. Different from existing pooling methods that use a fixed pooling norm, we propose to learn the norm in an end-to-end manner to automatically find the optimal ones for text representation in different tasks. In addition, we propose two methods to ensure the numerical stability of the model training. The first one is scale limiting, which re-scales the input to ensure non-negativity and alleviate the risk of exponential explosion. The second one is re-formulation, which decomposes the exponent operation to avoid computing the real-valued powers of the input and further accelerate the pooling operation. Experimental results on four benchmark datasets show that our approach can effectively improve the performance of attentive pooling.
Anthology ID:
2020.acl-main.267
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Editors:
Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2961–2970
Language:
URL:
https://aclanthology.org/2020.acl-main.267
DOI:
10.18653/v1/2020.acl-main.267
Bibkey:
Cite (ACL):
Chuhan Wu, Fangzhao Wu, Tao Qi, Xiaohui Cui, and Yongfeng Huang. 2020. Attentive Pooling with Learnable Norms for Text Representation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2961–2970, Online. Association for Computational Linguistics.
Cite (Informal):
Attentive Pooling with Learnable Norms for Text Representation (Wu et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-main.267.pdf
Video:
 http://slideslive.com/38929028