Universal Language Model Fine-tuning for Text Classification

Jeremy Howard, Sebastian Ruder


Abstract
Inductive transfer learning has greatly impacted computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch. We propose Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduce techniques that are key for fine-tuning a language model. Our method significantly outperforms the state-of-the-art on six text classification tasks, reducing the error by 18-24% on the majority of datasets. Furthermore, with only 100 labeled examples, it matches the performance of training from scratch on 100 times more data. We open-source our pretrained models and code.
Anthology ID:
P18-1031
Volume:
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
July
Year:
2018
Address:
Melbourne, Australia
Editors:
Iryna Gurevych, Yusuke Miyao
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
328–339
Language:
URL:
https://aclanthology.org/P18-1031
DOI:
10.18653/v1/P18-1031
Bibkey:
Cite (ACL):
Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics.
Cite (Informal):
Universal Language Model Fine-tuning for Text Classification (Howard & Ruder, ACL 2018)
Copy Citation:
PDF:
https://aclanthology.org/P18-1031.pdf
Poster:
 P18-1031.Poster.pdf
Code
 fastai/fastai +  additional community code
Data
AG NewsDBpediaIMDb Movie ReviewsWikiText-103WikiText-2Yelp