<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kumar-2026-looking">
    <titleInfo>
        <title>Looking Complete Is Not Being Grounded: Evaluating LLM Sitreps Against Human-Authored Situation Reports</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Sachin</namePart>
        <namePart type="family">Kumar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the First Workshop on Language Technologies for Crisis Preparedness and Response</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Belu</namePart>
            <namePart type="family">Ticona</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Fei</namePart>
            <namePart type="family">Xia</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">William</namePart>
            <namePart type="given">D</namePart>
            <namePart type="family">Lewis</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Antonios</namePart>
            <namePart type="family">Anastasopoulos</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Hengqin, China</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-450-7</identifier>
    </relatedItem>
    <abstract>Evaluation for automatic crisis reporting should measure what practitioners need, not what is convenient to compare. Through LLM-assisted, author-approved content analysis, we derive a 15-slot information schema from professional humanitarian situation reports using qualitative content analysis, then evaluate 121 human and 261 LLM-generated reports across five disasters and three hazard types. Under google/gemini-3.7-flash on Cyclone Idai, the best machine arm is nominally above the empirical human ceiling (0.827 vs. 0.821), but the difference is statistically indistinguishable from zero (95% bootstrap CI [-.049,.059]; permutation p=.852), and the ordering does not replicate under openai/gpt-5.6-luna. On a blind 100-item Idai sample, the author agrees with the Gemini judge at 0.910 accuracy and unweighted Cohen’s ąppa=.799. All four judges clear the plan’s raw-agreement criterion of .7; only Gemini reaches .7 on chance-corrected ąppa. Across the three primary events, machine reports reproduce 5.4–11.0% of same-day official figures. Across those same events, raw exact-number matching links 14.3–19.6% of official figures and 85.4–90.1% of machine figures to the social stream; manual audit confirms 45/50 declared absences, 4/25 official matches, and 24/25 machine matches semantically. The issuing-provenance gap is positive in all 12 observed event–judge cells: report date, organisation, series, and authority were not supplied at professional-report coverage in any cell. On the three primary events, ROUGE-L is uninformative, BERTScore is negatively associated with completeness after multiplicity correction, and a generic LLM judge changes sign.</abstract>
    <identifier type="citekey">kumar-2026-looking</identifier>
    <location>
        <url>https://aclanthology.org/2026.lt4cpr-1.7/</url>
    </location>
    <part>
        <date>2026-11</date>
        <extent unit="page">
            <start>65</start>
            <end>77</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Looking Complete Is Not Being Grounded: Evaluating LLM Sitreps Against Human-Authored Situation Reports
%A Kumar, Sachin
%Y Ticona, Belu
%Y Xia, Fei
%Y Lewis, William D.
%Y Anastasopoulos, Antonios
%S Proceedings of the First Workshop on Language Technologies for Crisis Preparedness and Response
%D 2026
%8 November
%I Association for Computational Linguistics
%C Hengqin, China
%@ 979-8-89176-450-7
%F kumar-2026-looking
%X Evaluation for automatic crisis reporting should measure what practitioners need, not what is convenient to compare. Through LLM-assisted, author-approved content analysis, we derive a 15-slot information schema from professional humanitarian situation reports using qualitative content analysis, then evaluate 121 human and 261 LLM-generated reports across five disasters and three hazard types. Under google/gemini-3.7-flash on Cyclone Idai, the best machine arm is nominally above the empirical human ceiling (0.827 vs. 0.821), but the difference is statistically indistinguishable from zero (95% bootstrap CI [-.049,.059]; permutation p=.852), and the ordering does not replicate under openai/gpt-5.6-luna. On a blind 100-item Idai sample, the author agrees with the Gemini judge at 0.910 accuracy and unweighted Cohen’s ąppa=.799. All four judges clear the plan’s raw-agreement criterion of .7; only Gemini reaches .7 on chance-corrected ąppa. Across the three primary events, machine reports reproduce 5.4–11.0% of same-day official figures. Across those same events, raw exact-number matching links 14.3–19.6% of official figures and 85.4–90.1% of machine figures to the social stream; manual audit confirms 45/50 declared absences, 4/25 official matches, and 24/25 machine matches semantically. The issuing-provenance gap is positive in all 12 observed event–judge cells: report date, organisation, series, and authority were not supplied at professional-report coverage in any cell. On the three primary events, ROUGE-L is uninformative, BERTScore is negatively associated with completeness after multiplicity correction, and a generic LLM judge changes sign.
%U https://aclanthology.org/2026.lt4cpr-1.7/
%P 65-77

Markdown (Informal)

[Looking Complete Is Not Being Grounded: Evaluating LLM Sitreps Against Human-Authored Situation Reports](https://aclanthology.org/2026.lt4cpr-1.7/) (Kumar, LT4CPR 2026)

• Looking Complete Is Not Being Grounded: Evaluating LLM Sitreps Against Human-Authored Situation Reports (Kumar, LT4CPR 2026)

ACL

• Sachin Kumar. 2026. Looking Complete Is Not Being Grounded: Evaluating LLM Sitreps Against Human-Authored Situation Reports. In Proceedings of the First Workshop on Language Technologies for Crisis Preparedness and Response, pages 65–77, Hengqin, China. Association for Computational Linguistics.