Attribute Injection for Pretrained Language Models: A New Benchmark and an Efficient Method

Reinald Kim Amplayo, Kang Min Yoo, Sang-Woo Lee


Abstract
Metadata attributes (e.g., user and product IDs from reviews) can be incorporated as additional inputs to neural-based NLP models, by expanding the architecture of the models to improve performance. However, recent models rely on pretrained language models (PLMs), in which previously used techniques for attribute injection are either nontrivial or cost-ineffective. In this paper, we introduce a benchmark for evaluating attribute injection models, which comprises eight datasets across a diverse range of tasks and domains and six synthetically sparsified ones. We also propose a lightweight and memory-efficient method to inject attributes into PLMs. We extend adapters, i.e. tiny plug-in feed-forward modules, to include attributes both independently of or jointly with the text. We use approximation techniques to parameterize the model efficiently for domains with large attribute vocabularies, and training mechanisms to handle multi-labeled and sparse attributes. Extensive experiments and analyses show that our method outperforms previous attribute injection methods and achieves state-of-the-art performance on all datasets.
Anthology ID:
2022.coling-1.88
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
1051–1064
Language:
URL:
https://aclanthology.org/2022.coling-1.88
DOI:
Bibkey:
Cite (ACL):
Reinald Kim Amplayo, Kang Min Yoo, and Sang-Woo Lee. 2022. Attribute Injection for Pretrained Language Models: A New Benchmark and an Efficient Method. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1051–1064, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Attribute Injection for Pretrained Language Models: A New Benchmark and an Efficient Method (Amplayo et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.88.pdf
Code
 rktamplayo/injector