Soft Representation Learning for Sparse Transfer

Haeju Park, Jinyoung Yeo, Gengyu Wang, Seung-won Hwang


Abstract
Transfer learning is effective for improving the performance of tasks that are related, and Multi-task learning (MTL) and Cross-lingual learning (CLL) are important instances. This paper argues that hard-parameter sharing, of hard-coding layers shared across different tasks or languages, cannot generalize well, when sharing with a loosely related task. Such case, which we call sparse transfer, might actually hurt performance, a phenomenon known as negative transfer. Our contribution is using adversarial training across tasks, to “soft-code” shared and private spaces, to avoid the shared space gets too sparse. In CLL, our proposed architecture considers another challenge of dealing with low-quality input.
Anthology ID:
P19-1151
Volume:
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2019
Address:
Florence, Italy
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1560–1568
Language:
URL:
https://aclanthology.org/P19-1151
DOI:
10.18653/v1/P19-1151
Bibkey:
Cite (ACL):
Haeju Park, Jinyoung Yeo, Gengyu Wang, and Seung-won Hwang. 2019. Soft Representation Learning for Sparse Transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1560–1568, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
Soft Representation Learning for Sparse Transfer (Park et al., ACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/P19-1151.pdf
Data
SNLI