<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="beau-crabbe-2024-codeinsight">
    <titleInfo>
        <title>CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Nathanaël</namePart>
        <namePart type="family">Beau</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Benoit</namePart>
        <namePart type="family">Crabbé</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Lun-Wei</namePart>
            <namePart type="family">Ku</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Andre</namePart>
            <namePart type="family">Martins</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Vivek</namePart>
            <namePart type="family">Srikumar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Bangkok, Thailand</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>We introduce a novel dataset tailored for code generation, aimed at aiding developers in common tasks. Our dataset provides examples that include a clarified intent, code snippets associated, and an average of three related unit tests. It encompasses a range of libraries such as Pandas, Numpy, and Regex, along with more than 70 standard libraries in Python code derived from Stack Overflow. Comprising 3,402 crafted examples by Python experts, our dataset is designed for both model finetuning and standalone evaluation. To complete unit tests evaluation, we categorize examples in order to get more fine grained analysis, enhancing the understanding of models’ strengths and weaknesses in specific coding tasks. The examples have been refined to reduce data contamination, a process confirmed by the performance of three leading models: Mistral 7B, CodeLLAMA 13B, and Starcoder 15B. We further investigate data-contamination testing GPT-4 performance on a part of our dataset. The benchmark can be accessed at anonymized address.</abstract>
    <identifier type="citekey">beau-crabbe-2024-codeinsight</identifier>
    <identifier type="doi">10.18653/v1/2024.findings-acl.354</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-acl.354</url>
    </location>
    <part>
        <date>2024-08</date>
        <extent unit="page">
            <start>5935</start>
            <end>5947</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow
%A Beau, Nathanaël
%A Crabbé, Benoit
%Y Ku, Lun-Wei
%Y Martins, Andre
%Y Srikumar, Vivek
%S Findings of the Association for Computational Linguistics: ACL 2024
%D 2024
%8 August
%I Association for Computational Linguistics
%C Bangkok, Thailand
%F beau-crabbe-2024-codeinsight
%X We introduce a novel dataset tailored for code generation, aimed at aiding developers in common tasks. Our dataset provides examples that include a clarified intent, code snippets associated, and an average of three related unit tests. It encompasses a range of libraries such as Pandas, Numpy, and Regex, along with more than 70 standard libraries in Python code derived from Stack Overflow. Comprising 3,402 crafted examples by Python experts, our dataset is designed for both model finetuning and standalone evaluation. To complete unit tests evaluation, we categorize examples in order to get more fine grained analysis, enhancing the understanding of models’ strengths and weaknesses in specific coding tasks. The examples have been refined to reduce data contamination, a process confirmed by the performance of three leading models: Mistral 7B, CodeLLAMA 13B, and Starcoder 15B. We further investigate data-contamination testing GPT-4 performance on a part of our dataset. The benchmark can be accessed at anonymized address.
%R 10.18653/v1/2024.findings-acl.354
%U https://aclanthology.org/2024.findings-acl.354
%U https://doi.org/10.18653/v1/2024.findings-acl.354
%P 5935-5947

Markdown (Informal)

[CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow](https://aclanthology.org/2024.findings-acl.354) (Beau & Crabbé, Findings 2024)

• CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow (Beau & Crabbé, Findings 2024)

ACL

• Nathanaël Beau and Benoit Crabbé. 2024. CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5935–5947, Bangkok, Thailand. Association for Computational Linguistics.