The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders

Han He, Jinho D. Choi


Abstract
Multi-task learning with transformer encoders (MTL) has emerged as a powerful technique to improve performance on closely-related tasks for both accuracy and efficiency while a question still remains whether or not it would perform as well on tasks that are distinct in nature. We first present MTL results on five NLP tasks, POS, NER, DEP, CON, and SRL, and depict its deficiency over single-task learning. We then conduct an extensive pruning analysis to show that a certain set of attention heads get claimed by most tasks during MTL, who interfere with one another to fine-tune those heads for their own objectives. Based on this finding, we propose the Stem Cell Hypothesis to reveal the existence of attention heads naturally talented for many tasks that cannot be jointly trained to create adequate embeddings for all of those tasks. Finally, we design novel parameter-free probes to justify our hypothesis and demonstrate how attention heads are transformed across the five tasks during MTL through label analysis.
Anthology ID:
2021.emnlp-main.451
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5555–5577
Language:
URL:
https://aclanthology.org/2021.emnlp-main.451
DOI:
10.18653/v1/2021.emnlp-main.451
Bibkey:
Cite (ACL):
Han He and Jinho D. Choi. 2021. The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5555–5577, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders (He & Choi, EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.451.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.451.mp4
Code
 emorynlp/stem-cell-hypothesis