CodeQA: A Question Answering Dataset for Source Code Comprehension

Chenxiao Liu, Xiaojun Wan


Abstract
We propose CodeQA, a free-form question answering dataset for the purpose of source code comprehension: given a code snippet and a question, a textual answer is required to be generated. CodeQA contains a Java dataset with 119,778 question-answer pairs and a Python dataset with 70,085 question-answer pairs. To obtain natural and faithful questions and answers, we implement syntactic rules and semantic analysis to transform code comments into question-answer pairs. We present the construction process and conduct systematic analysis of our dataset. Experiment results achieved by several neural baselines on our dataset are shown and discussed. While research on question-answering and machine reading comprehension develops rapidly, few prior work has drawn attention to code question answering. This new dataset can serve as a useful research benchmark for source code comprehension.
Anthology ID:
2021.findings-emnlp.223
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2021
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
2618–2632
Language:
URL:
https://aclanthology.org/2021.findings-emnlp.223
DOI:
10.18653/v1/2021.findings-emnlp.223
Bibkey:
Cite (ACL):
Chenxiao Liu and Xiaojun Wan. 2021. CodeQA: A Question Answering Dataset for Source Code Comprehension. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2618–2632, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
CodeQA: A Question Answering Dataset for Source Code Comprehension (Liu & Wan, Findings 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.findings-emnlp.223.pdf
Video:
 https://aclanthology.org/2021.findings-emnlp.223.mp4
Code
 jadecxliu/codeqa
Data
CodeQAMS MARCO