Bag-of-Words Baselines for Semantic Code Search

Xinyu Zhang, Ji Xin, Andrew Yates, Jimmy Lin


Abstract
The task of semantic code search is to retrieve code snippets from a source code corpus based on an information need expressed in natural language. The semantic gap between natural language and programming languages has for long been regarded as one of the most significant obstacles to the effectiveness of keyword-based information retrieval (IR) methods. It is a common assumption that “traditional” bag-of-words IR methods are poorly suited for semantic code search: our work empirically investigates this assumption. Specifically, we examine the effectiveness of two traditional IR methods, namely BM25 and RM3, on the CodeSearchNet Corpus, which consists of natural language queries paired with relevant code snippets. We find that the two keyword-based methods outperform several pre-BERT neural models. We also compare several code-specific data pre-processing strategies and find that specialized tokenization improves effectiveness.
Anthology ID:
2021.nlp4prog-1.10
Volume:
Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021)
Month:
August
Year:
2021
Address:
Online
Editors:
Royi Lachmy, Ziyu Yao, Greg Durrett, Milos Gligoric, Junyi Jessy Li, Ray Mooney, Graham Neubig, Yu Su, Huan Sun, Reut Tsarfaty
Venue:
NLP4Prog
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
88–94
Language:
URL:
https://aclanthology.org/2021.nlp4prog-1.10
DOI:
10.18653/v1/2021.nlp4prog-1.10
Bibkey:
Cite (ACL):
Xinyu Zhang, Ji Xin, Andrew Yates, and Jimmy Lin. 2021. Bag-of-Words Baselines for Semantic Code Search. In Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021), pages 88–94, Online. Association for Computational Linguistics.
Cite (Informal):
Bag-of-Words Baselines for Semantic Code Search (Zhang et al., NLP4Prog 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.nlp4prog-1.10.pdf
Data
CodeSearchNet