Making Parameter-efficient Tuning More Efficient: A Unified Framework for Classification Tasks

Xin Zhou, Ruotian Ma, Yicheng Zou, Xuanting Chen, Tao Gui, Qi Zhang, Xuanjing Huang, Rui Xie, Wei Wu


Abstract
Large pre-trained language models (PLMs) have demonstrated superior performance in industrial applications. Recent studies have explored parameter-efficient PLM tuning, which only updates a small amount of task-specific parameters while achieving both high efficiency and comparable performance against standard fine-tuning. However, all these methods ignore the inefficiency problem caused by the task-specific output layers, which is inflexible for us to re-use PLMs and introduces non-negligible parameters. In this work, we focus on the text classification task and propose plugin-tuning, a framework that further improves the efficiency of existing parameter-efficient methods with a unified classifier. Specifically, we re-formulate both token and sentence classification tasks into a unified language modeling task, and map label spaces of different tasks into the same vocabulary space. In this way, we can directly re-use the language modeling heads of PLMs, avoiding introducing extra parameters for different tasks. We conduct experiments on six classification benchmarks. The experimental results show that plugin-tuning can achieve comparable performance against fine-tuned PLMs, while further saving around 50% parameters on top of other parameter-efficient methods.
Anthology ID:
2022.coling-1.615
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
7053–7064
Language:
URL:
https://aclanthology.org/2022.coling-1.615
DOI:
Bibkey:
Cite (ACL):
Xin Zhou, Ruotian Ma, Yicheng Zou, Xuanting Chen, Tao Gui, Qi Zhang, Xuanjing Huang, Rui Xie, and Wei Wu. 2022. Making Parameter-efficient Tuning More Efficient: A Unified Framework for Classification Tasks. In Proceedings of the 29th International Conference on Computational Linguistics, pages 7053–7064, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Making Parameter-efficient Tuning More Efficient: A Unified Framework for Classification Tasks (Zhou et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.615.pdf
Code
 xzhou20/plugin-tuning
Data
CoNLL-2003GLUEPenn Treebank