You Only Need Attention to Traverse Trees

Mahtab Ahmed, Muhammad Rifayat Samee, Robert E. Mercer


Abstract
In recent NLP research, a topic of interest is universal sentence encoding, sentence representations that can be used in any supervised task. At the word sequence level, fully attention-based models suffer from two problems: a quadratic increase in memory consumption with respect to the sentence length and an inability to capture and use syntactic information. Recursive neural nets can extract very good syntactic information by traversing a tree structure. To this end, we propose Tree Transformer, a model that captures phrase level syntax for constituency trees as well as word-level dependencies for dependency trees by doing recursive traversal only with attention. Evaluation of this model on four tasks gets noteworthy results compared to the standard transformer and LSTM-based models as well as tree-structured LSTMs. Ablation studies to find whether positional information is inherently encoded in the trees and which type of attention is suitable for doing the recursive traversal are provided.
Anthology ID:
P19-1030
Volume:
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2019
Address:
Florence, Italy
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
316–322
Language:
URL:
https://aclanthology.org/P19-1030
DOI:
10.18653/v1/P19-1030
Bibkey:
Cite (ACL):
Mahtab Ahmed, Muhammad Rifayat Samee, and Robert E. Mercer. 2019. You Only Need Attention to Traverse Trees. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 316–322, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
You Only Need Attention to Traverse Trees (Ahmed et al., ACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/P19-1030.pdf
Video:
 https://vimeo.com/384000960
Data
SST