Learning to Retrieve Passages without Supervision

Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, Amir Globerson


Abstract
Dense retrievers for open-domain question answering (ODQA) have been shown to achieve impressive performance by training on large datasets of question-passage pairs. In this work we ask whether this dependence on labeled data can be reduced via unsupervised pretraining that is geared towards ODQA. We show this is in fact possible, via a novel pretraining scheme designed for retrieval. Our “recurring span retrieval” approach uses recurring spans across passages in a document to create pseudo examples for contrastive learning. Our pretraining scheme directly controls for term overlap across pseudo queries and relevant passages, thus allowing to model both lexical and semantic relations between them. The resulting model, named Spider, performs surprisingly well without any labeled training examples on a wide range of ODQA datasets. Specifically, it significantly outperforms all other pretrained baselines in a zero-shot setting, and is competitive with BM25, a strong sparse baseline. Moreover, a hybrid retriever over Spider and BM25 improves over both, and is often competitive with DPR models, which are trained on tens of thousands of examples. Last, notable gains are observed when using Spider as an initialization for supervised training.
Anthology ID:
2022.naacl-main.193
Volume:
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
July
Year:
2022
Address:
Seattle, United States
Editors:
Marine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2687–2700
Language:
URL:
https://aclanthology.org/2022.naacl-main.193
DOI:
10.18653/v1/2022.naacl-main.193
Bibkey:
Cite (ACL):
Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, and Amir Globerson. 2022. Learning to Retrieve Passages without Supervision. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2687–2700, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
Learning to Retrieve Passages without Supervision (Ram et al., NAACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.naacl-main.193.pdf
Software:
 2022.naacl-main.193.software.zip
Video:
 https://aclanthology.org/2022.naacl-main.193.mp4
Code
 oriram/spider
Data
EntityQuestionsNatural QuestionsSQuADTriviaQA