<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="naiman-2026-star">
    <titleInfo>
        <title>Star Light, Star Bright: Can Multimodal Models Parse an Astronomy Visual Question Answering Dataset Right?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">J</namePart>
        <namePart type="given">P</namePart>
        <namePart type="family">Naiman</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 4th Workshop on Artificial Intelligence for Scientific Publications</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Alberto</namePart>
            <namePart type="family">Accomazzi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tirthankar</namePart>
            <namePart type="family">Ghosal</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Felix</namePart>
            <namePart type="family">Grezes</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kelly</namePart>
            <namePart type="family">Lockhart</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Atilla</namePart>
            <namePart type="given">Kaan</namePart>
            <namePart type="family">Alkan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Hengqin, Zhuhai, China</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-456-9</identifier>
    </relatedItem>
    <abstract>Figures are essential for scientific communication, and the field of Visual Question Answering (VQA) aims to test the figure understanding capabilities of large multimodal models (LMMs). Here, we investigate the accuracy of several low-cost LMMs on domain-specific VQA tasks for astronomical images of the sky. As instrument for this task we build a domain-specific VQA dataset comprising 2001 partially-synthetic figures divided evenly between contour plots, real telescope images drawn from NASA’s SkyView, and “fake” sky images which share the axes and angular scales of the real images but draw their colors from a gaussian mixture model, alongside 47357 questions. Each figure is released as pristine and with realistic page-aging degradations mimicking the historical astronomical corpus. Testing three low-cost proprietary LMMs, we find models frequently fail to answer domain-specific questions at high accuracy. An initial comparison with 104 human-verified, astrometrically tagged figures from historical journal publications shows similar median errors in angular extent. All code is available on GitHub: https://github.com/jnaiman/SkyImagesWASP2026 and dataset is available on HuggingFace: https://huggingface.co/datasets/ReadingTimeMachine/astro_sky_image_vqa.</abstract>
    <identifier type="citekey">naiman-2026-star</identifier>
    <location>
        <url>https://aclanthology.org/2026.wasp-1.20/</url>
    </location>
    <part>
        <date>2026-11</date>
        <extent unit="page">
            <start>180</start>
            <end>191</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Star Light, Star Bright: Can Multimodal Models Parse an Astronomy Visual Question Answering Dataset Right?
%A Naiman, J. P.
%Y Accomazzi, Alberto
%Y Ghosal, Tirthankar
%Y Grezes, Felix
%Y Lockhart, Kelly
%Y Alkan, Atilla Kaan
%S Proceedings of the 4th Workshop on Artificial Intelligence for Scientific Publications
%D 2026
%8 November
%I Association for Computational Linguistics
%C Hengqin, Zhuhai, China
%@ 979-8-89176-456-9
%F naiman-2026-star
%X Figures are essential for scientific communication, and the field of Visual Question Answering (VQA) aims to test the figure understanding capabilities of large multimodal models (LMMs). Here, we investigate the accuracy of several low-cost LMMs on domain-specific VQA tasks for astronomical images of the sky. As instrument for this task we build a domain-specific VQA dataset comprising 2001 partially-synthetic figures divided evenly between contour plots, real telescope images drawn from NASA’s SkyView, and “fake” sky images which share the axes and angular scales of the real images but draw their colors from a gaussian mixture model, alongside 47357 questions. Each figure is released as pristine and with realistic page-aging degradations mimicking the historical astronomical corpus. Testing three low-cost proprietary LMMs, we find models frequently fail to answer domain-specific questions at high accuracy. An initial comparison with 104 human-verified, astrometrically tagged figures from historical journal publications shows similar median errors in angular extent. All code is available on GitHub: https://github.com/jnaiman/SkyImagesWASP2026 and dataset is available on HuggingFace: https://huggingface.co/datasets/ReadingTimeMachine/astro_sky_image_vqa.
%U https://aclanthology.org/2026.wasp-1.20/
%P 180-191

Markdown (Informal)

[Star Light, Star Bright: Can Multimodal Models Parse an Astronomy Visual Question Answering Dataset Right?](https://aclanthology.org/2026.wasp-1.20/) (Naiman, WASP 2026)

• Star Light, Star Bright: Can Multimodal Models Parse an Astronomy Visual Question Answering Dataset Right? (Naiman, WASP 2026)

ACL

• J. P. Naiman. 2026. Star Light, Star Bright: Can Multimodal Models Parse an Astronomy Visual Question Answering Dataset Right?. In Proceedings of the 4th Workshop on Artificial Intelligence for Scientific Publications, pages 180–191, Hengqin, Zhuhai, China. Association for Computational Linguistics.