The Inefficiency of Language Models in Scholarly Retrieval: An Experimental Walk-through

Shruti Singh, Mayank Singh


Abstract
Language models are increasingly becoming popular in AI-powered scientific IR systems. This paper evaluates popular scientific language models in handling (i) short-query texts and (ii) textual neighbors. Our experiments showcase the inability to retrieve relevant documents for a short-query text even under the most relaxed conditions. Additionally, we leverage textual neighbors, generated by small perturbations to the original text, to demonstrate that not all perturbations lead to close neighbors in the embedding space. Further, an exhaustive categorization yields several classes of orthographically and semantically related, partially related and completely unrelated neighbors. Retrieval performance turns out to be more influenced by the surface form rather than the semantics of the text.
Anthology ID:
2022.findings-acl.249
Volume:
Findings of the Association for Computational Linguistics: ACL 2022
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3153–3173
Language:
URL:
https://aclanthology.org/2022.findings-acl.249
DOI:
10.18653/v1/2022.findings-acl.249
Bibkey:
Cite (ACL):
Shruti Singh and Mayank Singh. 2022. The Inefficiency of Language Models in Scholarly Retrieval: An Experimental Walk-through. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3153–3173, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
The Inefficiency of Language Models in Scholarly Retrieval: An Experimental Walk-through (Singh & Singh, Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-acl.249.pdf
Code
 shruti-singh/scilm_exp