HumMusQA: A Human-written Music Understanding QA Benchmark Dataset

Benno Weck; Pablo Puentes; Andrea Poltronieri; Satyajeet Prabhu; Dmitry Bogdanov

HumMusQA: A Human-written Music Understanding QA Benchmark Dataset

Benno Weck, Pablo Puentes, Andrea Poltronieri, Satyajeet Prabhu, Dmitry Bogdanov

Correct Metadata for

Use this form to create a GitHub issue with structured data describing the correction. You will need a GitHub account. Once you create that issue, the correction will be reviewed by a staff member.

⚠️ Mobile Users: Submitting this form to create a new issue will only work with github.com, not the GitHub Mobile app.

Important: The Anthology treat PDFs as authoritative. Please use this form only to correct data that is out of line with the PDF. See our corrections guidelines if you need to change the PDF.

Title Adjust the title. Retain tags such as <fixed-case>.

Authors Adjust author names and order to match the PDF.

Abstract Correct abstract if needed. Retain XML formatting tags such as <tex-math>. You may use <b>...</b> for bold, <i>...</i> for italic, and <url>...</url> for URLs.

Verification against PDF Ensure that the new title/authors match the snapshot below. (If there is no snapshot or it is too small, consult the PDF.)

Authors concatenated from the text boxes above:

ALL author names match the snapshot above—including middle initials, hyphens, and accents.

Abstract

The evaluation of music understanding in Large Audio-Language Models (LALMs) requires a rigorously defined benchmark that truly tests whether models can perceive and interpret music, a standard that current data methodologies frequently fail to meet.This paper introduces a meticulously structured approach to music evaluation, proposing a new dataset of 320 hand-written questions curated and validated by experts with musical training, arguing that such focused, manual curation is superior for probing complex audio comprehension.To demonstrate the use of the dataset, we benchmark six state-of-the-art LALMs and additionally test their robustness to uni-modal shortcuts.

Anthology ID:: 2026.nlp4musa-1.9
Volume:: Proceedings of the 4th Workshop on NLP for Music and Audio (NLP4MusA 2026)
Month:: March
Year:: 2026
Address:: Rabat, Morocco
Editors:: Elena V. Epure, Sergio Oramas, SeungHeon Doh, Pedro Ramoneda, Anna Kruspe, Mohamed Sordo
Venues:: NLP4MusA | WS
SIG:
Publisher:: Association for Computational Linguistics
Note:
Pages:: 58–67
Language:
URL:: https://aclanthology.org/2026.nlp4musa-1.9/
DOI:
Bibkey:
Cite (ACL):: Benno Weck, Pablo Puentes, Andrea Poltronieri, Satyajeet Prabhu, and Dmitry Bogdanov. 2026. HumMusQA: A Human-written Music Understanding QA Benchmark Dataset. In Proceedings of the 4th Workshop on NLP for Music and Audio (NLP4MusA 2026), pages 58–67, Rabat, Morocco. Association for Computational Linguistics.
Cite (Informal):: HumMusQA: A Human-written Music Understanding QA Benchmark Dataset (Weck et al., NLP4MusA 2026)
Copy Citation:
PDF:: https://aclanthology.org/2026.nlp4musa-1.9.pdf

PDF Cite Search Fix data

Export citation

BibTeX
MODS XML
Endnote
Preformatted

@inproceedings{weck-etal-2026-hummusqa,
    title = "{H}um{M}us{QA}: A Human-written Music Understanding {QA} Benchmark Dataset",
    author = "Weck, Benno  and
      Puentes, Pablo  and
      Poltronieri, Andrea  and
      Prabhu, Satyajeet  and
      Bogdanov, Dmitry",
    editor = "Epure, Elena V.  and
      Oramas, Sergio  and
      Doh, SeungHeon  and
      Ramoneda, Pedro  and
      Kruspe, Anna  and
      Sordo, Mohamed",
    booktitle = "Proceedings of the 4th Workshop on {NLP} for Music and Audio ({NLP}4{M}us{A} 2026)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.nlp4musa-1.9/",
    pages = "58--67",
    ISBN = "979-8-89176-369-2",
    abstract = "The evaluation of music understanding in Large Audio-Language Models (LALMs) requires a rigorously defined benchmark that truly tests whether models can perceive and interpret music, a standard that current data methodologies frequently fail to meet.This paper introduces a meticulously structured approach to music evaluation, proposing a new dataset of 320 hand-written questions curated and validated by experts with musical training, arguing that such focused, manual curation is superior for probing complex audio comprehension.To demonstrate the use of the dataset, we benchmark six state-of-the-art LALMs and additionally test their robustness to uni-modal shortcuts."
}

Download as File

<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="weck-etal-2026-hummusqa">
    <titleInfo>
        <title>HumMusQA: A Human-written Music Understanding QA Benchmark Dataset</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Benno</namePart>
        <namePart type="family">Weck</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Pablo</namePart>
        <namePart type="family">Puentes</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Andrea</namePart>
        <namePart type="family">Poltronieri</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Satyajeet</namePart>
        <namePart type="family">Prabhu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Dmitry</namePart>
        <namePart type="family">Bogdanov</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-03</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 4th Workshop on NLP for Music and Audio (NLP4MusA 2026)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Elena</namePart>
            <namePart type="given">V</namePart>
            <namePart type="family">Epure</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sergio</namePart>
            <namePart type="family">Oramas</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">SeungHeon</namePart>
            <namePart type="family">Doh</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Pedro</namePart>
            <namePart type="family">Ramoneda</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Anna</namePart>
            <namePart type="family">Kruspe</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mohamed</namePart>
            <namePart type="family">Sordo</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Rabat, Morocco</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-369-2</identifier>
    </relatedItem>
    <abstract>The evaluation of music understanding in Large Audio-Language Models (LALMs) requires a rigorously defined benchmark that truly tests whether models can perceive and interpret music, a standard that current data methodologies frequently fail to meet.This paper introduces a meticulously structured approach to music evaluation, proposing a new dataset of 320 hand-written questions curated and validated by experts with musical training, arguing that such focused, manual curation is superior for probing complex audio comprehension.To demonstrate the use of the dataset, we benchmark six state-of-the-art LALMs and additionally test their robustness to uni-modal shortcuts.</abstract>
    <identifier type="citekey">weck-etal-2026-hummusqa</identifier>
    <location>
        <url>https://aclanthology.org/2026.nlp4musa-1.9/</url>
    </location>
    <part>
        <date>2026-03</date>
        <extent unit="page">
            <start>58</start>
            <end>67</end>
        </extent>
    </part>
</mods>
</modsCollection>

Download as File

%0 Conference Proceedings
%T HumMusQA: A Human-written Music Understanding QA Benchmark Dataset
%A Weck, Benno
%A Puentes, Pablo
%A Poltronieri, Andrea
%A Prabhu, Satyajeet
%A Bogdanov, Dmitry
%Y Epure, Elena V.
%Y Oramas, Sergio
%Y Doh, SeungHeon
%Y Ramoneda, Pedro
%Y Kruspe, Anna
%Y Sordo, Mohamed
%S Proceedings of the 4th Workshop on NLP for Music and Audio (NLP4MusA 2026)
%D 2026
%8 March
%I Association for Computational Linguistics
%C Rabat, Morocco
%@ 979-8-89176-369-2
%F weck-etal-2026-hummusqa
%X The evaluation of music understanding in Large Audio-Language Models (LALMs) requires a rigorously defined benchmark that truly tests whether models can perceive and interpret music, a standard that current data methodologies frequently fail to meet.This paper introduces a meticulously structured approach to music evaluation, proposing a new dataset of 320 hand-written questions curated and validated by experts with musical training, arguing that such focused, manual curation is superior for probing complex audio comprehension.To demonstrate the use of the dataset, we benchmark six state-of-the-art LALMs and additionally test their robustness to uni-modal shortcuts.
%U https://aclanthology.org/2026.nlp4musa-1.9/
%P 58-67

Download as File

Markdown (Informal)

[HumMusQA: A Human-written Music Understanding QA Benchmark Dataset](https://aclanthology.org/2026.nlp4musa-1.9/) (Weck et al., NLP4MusA 2026)

HumMusQA: A Human-written Music Understanding QA Benchmark Dataset (Weck et al., NLP4MusA 2026)

ACL

Benno Weck, Pablo Puentes, Andrea Poltronieri, Satyajeet Prabhu, and Dmitry Bogdanov. 2026. HumMusQA: A Human-written Music Understanding QA Benchmark Dataset. In Proceedings of the 4th Workshop on NLP for Music and Audio (NLP4MusA 2026), pages 58–67, Rabat, Morocco. Association for Computational Linguistics.