Coarse-to-Fine: Hierarchical Multi-task Learning for Natural Language Understanding

Zhaoye Fei, Yu Tian, Yongkang Wu, Xinyu Zhang, Yutao Zhu, Zheng Liu, Jiawen Wu, Dejiang Kong, Ruofei Lai, Zhao Cao, Zhicheng Dou, Xipeng Qiu


Abstract
Generalized text representations are the foundation of many natural language understanding tasks. To fully utilize the different corpus, it is inevitable that models need to understand the relevance among them. However, many methods ignore the relevance and adopt a single-channel model (a coarse paradigm) directly for all tasks, which lacks enough rationality and interpretation. In addition, some existing works learn downstream tasks by stitches skill block (a fine paradigm), which might cause irrational results due to its redundancy and noise. In this work, we first analyze the task correlation through three different perspectives, , data property, manual design, and model-based relevance, based on which the similar tasks are grouped together. Then, we propose a hierarchical framework with a coarse-to-fine paradigm, with the bottom level shared to all the tasks, the mid-level divided to different groups, and the top-level assigned to each of the tasks. This allows our model to learn basic language properties from all tasks, boost performance on relevant tasks, and reduce the negative impact from irrelevant tasks. Our experiments on 13 benchmark datasets across five natural language understanding tasks demonstrate the superiority of our method.
Anthology ID:
2022.coling-1.439
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
4952–4964
Language:
URL:
https://aclanthology.org/2022.coling-1.439
DOI:
Bibkey:
Cite (ACL):
Zhaoye Fei, Yu Tian, Yongkang Wu, Xinyu Zhang, Yutao Zhu, Zheng Liu, Jiawen Wu, Dejiang Kong, Ruofei Lai, Zhao Cao, Zhicheng Dou, and Xipeng Qiu. 2022. Coarse-to-Fine: Hierarchical Multi-task Learning for Natural Language Understanding. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4952–4964, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Coarse-to-Fine: Hierarchical Multi-task Learning for Natural Language Understanding (Fei et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.439.pdf
Data
BoolQGLUEIMDb Movie ReviewsMRPCMultiNLIMultiRCQNLISNLISSTSuperGLUE