%0 Conference Proceedings %T A Little Bit Is Worse Than None: Ranking with Limited Training Data %A Zhang, Xinyu %A Yates, Andrew %A Lin, Jimmy %Y Moosavi, Nafise Sadat %Y Fan, Angela %Y Shwartz, Vered %Y Glavaš, Goran %Y Joty, Shafiq %Y Wang, Alex %Y Wolf, Thomas %S Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing %D 2020 %8 November %I Association for Computational Linguistics %C Online %F zhang-etal-2020-little %X Researchers have proposed simple yet effective techniques for the retrieval problem based on using BERT as a relevance classifier to rerank initial candidates from keyword search. In this work, we tackle the challenge of fine-tuning these models for specific domains in a data and computationally efficient manner. Typically, researchers fine-tune models using corpus-specific labeled data from sources such as TREC. We first answer the question: How much data of this type do we need? Recognizing that the most computationally efficient training is no training, we explore zero-shot ranking using BERT models that have already been fine-tuned with the large MS MARCO passage retrieval dataset. We arrive at the surprising and novel finding that “some” labeled in-domain data can be worse than none at all. %R 10.18653/v1/2020.sustainlp-1.14 %U https://aclanthology.org/2020.sustainlp-1.14 %U https://doi.org/10.18653/v1/2020.sustainlp-1.14 %P 107-112