Ashutosh Ojha
Author directory2026
What Do Developers Ask? A Taxonomy of Code Related Questions
Ashutosh Ojha | Ashita Saxena | Varshini Reddy Pateel | Palanivel Kodeswaran | Sayandeep Sen | Srikanth G. Tamilselvam
Findings of the Association for Computational Linguistics: EMNLP 2026
Ashutosh Ojha | Ashita Saxena | Varshini Reddy Pateel | Palanivel Kodeswaran | Sayandeep Sen | Srikanth G. Tamilselvam
Findings of the Association for Computational Linguistics: EMNLP 2026
Progress in Code Question Answering (CodeQA) depends on benchmarks that reflect what developers actually ask. Yet most existing CodeQA datasets are built from docstring templates or GitHub issues, and rarely capture the diversity of real, repository-level developer queries. We address this gap by releasing a taxonomy of real-world code-assistance queries, grounded in 3,052 authentic user interactions from the RepoChat dataset. We construct the taxonomy via a semi-automated pipeline that combines clustering, LLM-based labeling, and an iterative refinement loop driven by inter-model disagreement, and we compare the result against three prior taxonomy-generation methods (TaxoGen, TaxoCom, TaxoAdapt). Our taxonomy achieves a favorable tradeoff between granularity and usability—86 nodes with 93.2% node relevance, 99.4% path validity, and 81.25% sibling coherence—while remaining concise enough for downstream use. We then apply the taxonomy in two ways: (i) auditing existing Code related benchmarks to surface systematically underrepresented query types, and (ii) generating a synthetic CodeQA dataset whose categories are guided by the taxonomy.