Yin Yang
2021
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT
Prakhar Ganesh
|
Yao Chen
|
Xin Lou
|
Mohammad Ali Khan
|
Yin Yang
|
Hassan Sajjad
|
Preslav Nakov
|
Deming Chen
|
Marianne Winslett
Transactions of the Association for Computational Linguistics, Volume 9
Pre-trained Transformer-based models have achieved state-of-the-art performance for various Natural Language Processing (NLP) tasks. However, these models often have billions of parameters, and thus are too resource- hungry and computation-intensive to suit low- capability devices or applications with strict latency requirements. One potential remedy for this is model compression, which has attracted considerable research attention. Here, we summarize the research in compressing Transformers, focusing on the especially popular BERT model. In particular, we survey the state of the art in compression for BERT, we clarify the current best practices for compressing large-scale Transformer models, and we provide insights into the workings of various methods. Our categorization and analysis also shed light on promising future research directions for achieving lightweight, accurate, and generic NLP models.
2019
Fine-Grained Propaganda Detection with Fine-Tuned BERT
Shehel Yoosuf
|
Yin Yang
Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda
This paper presents the winning solution of the Fragment Level Classification (FLC) task in the Fine Grained Propaganda Detection competition at the NLP4IF’19 workshop. The goal of the FLC task is to detect and classify textual segments that correspond to one of the 18 given propaganda techniques in a news articles dataset. The main idea of our solution is to perform word-level classification using fine-tuned BERT, a popular pre-trained language model. Besides presenting the model and its evaluation results, we also investigate the attention heads in the model, which provide insights into what the model learns, as well as aspects for potential improvements.
Search
Co-authors
- Prakhar Ganesh 1
- Yao Chen 1
- Xin Lou 1
- Mohammad Ali Khan 1
- Hassan Sajjad 1
- show all...