Recommendations for Datasets for Source Code Summarization

Alexander LeClair, Collin McMillan


Abstract
Source Code Summarization is the task of writing short, natural language descriptions of source code. The main use for these descriptions is in software documentation e.g. the one-sentence Java method descriptions in JavaDocs. Code summarization is rapidly becoming a popular research problem, but progress is restrained due to a lack of suitable datasets. In addition, a lack of community standards for creating datasets leads to confusing and unreproducible research results – we observe swings in performance of more than 33% due only to changes in dataset design. In this paper, we make recommendations for these standards from experimental results. We release a dataset based on prior work of over 2.1m pairs of Java methods and one sentence method descriptions from over 28k Java projects. We describe the dataset and point out key differences from natural language data, to guide and support future researchers.
Anthology ID:
N19-1394
Volume:
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
Month:
June
Year:
2019
Address:
Minneapolis, Minnesota
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3931–3937
Language:
URL:
https://aclanthology.org/N19-1394
DOI:
10.18653/v1/N19-1394
Bibkey:
Cite (ACL):
Alexander LeClair and Collin McMillan. 2019. Recommendations for Datasets for Source Code Summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3931–3937, Minneapolis, Minnesota. Association for Computational Linguistics.
Cite (Informal):
Recommendations for Datasets for Source Code Summarization (LeClair & McMillan, NAACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/N19-1394.pdf
Code
 additional community code
Data
Funcom