Large Language Models (LLMs) are increasingly explored in the AEC (Architecture, Engineering, and Construction) domain, yet their reliability in regulatory tasks remains uncertain.

In this study, we focused on Hong Kong fire safety codes: together with domain experts, we create a localized dataset of multiple-choice (MCQs) and open-ended questions (OEQs) to evaluate 12 widely used LLMs, including both proprietary and open-source models.

Our findings show that proprietary models may achieve 80–90% accuracy on MCQs, while open-source models generally lag behind. On OEQ, humans and LLM judges rated the answers at 50-70% accuracy, suggesting that LLMs in specialized compliance tasks still have large margins of improvement.

Communicating with the public during an emergency entails issuing warning messages promptly. In Europe, it often requires producing them in more than one language. Recent work has shown that language models can be trained to generate warning messages as a crisis unfolds, but the available data covers only English. To fill this gap, we present *CrisiLangs*, a multilingual extension of *CrisiText* (an English dataset of crisis events paired with warning messages written according to expert communication guidelines) to Italian, Bulgarian, German, and Greek. Expert native speakers translated the warning messages of 1,000 events drawn from 411 scenarios, for a total of 5,000 event-message pairs across five languages. We test the resource to fine-tune a multilingual model on all five languages jointly, with and without the emergency guidelines in the prompt, and compare it with a zero-shot baseline. The fine-tuned models outperform the baseline on ROUGE-L, BLEU and BERTScore in every language and configuration. In a human preference study, low decoding temperatures receive the most preferences in all five languages and in both configurations, while the effect of the guidelines varies by language.

Machine translation (MT) is increasingly utilized in multilingual crisis communication, yet conventional evaluation does not directly assess whether translated emergency messages preserve their actionability. We introduce SIREN (Situation-aware Index for Response and Emergency ActioNability), a human evaluation framework assessing whether translation choices compromise protective action, directive force, and urgency of the source text. We evaluate SIREN in two complementary settings. At the error level, we apply it to HOPE-annotated Italian–English translations produced by ChatGPT-4o and Google Translate: 31.63% and 48.73% of errors, respectively, compromise actionability, predominantly by affecting the protective action. At the message level, we evaluate English–French translations produced by TranslateGemma-4B and Google Translate, with actionability preserved in 85.34% and 92.67% of translations, respectively. TranslateGemma-4B failures predominantly weaken directive force, whereas Google Translate failures more often compromise protective actions and receive Major impact ratings. These findings show that translation systems can differ not only in how often actionability is lost, but also in the nature and functional impact of these failures. Alongside SIREN, we contribute an English–French crisis-translation resource with human actionability annotations and post-edited reference translations, supporting further research on function-oriented evaluation for crisis communication. Our experimental data is available at: https://github.com/mcstaiano/SIREN.

It is essential that we communicate with the entirety of an affected population during a crisis, including those who do not speak a region’s dominant language(s). Commercial Translation tools offer a tempting solutions to broaden access and can provide a lifeline when no other resources are available. However, Emergency response managers often operate in the dark regarding the quality and adequacy of these automated outputs. In this paper, we conduct a nationwide survey of emergency managers to understand their current workflows, perspectives, and operational challenges. We then evaluate commercial translation tools against these needs, shedding light on their utility and offering practical guidance on how and when these tools can be safely used in emergency response.

Evaluation for automatic crisis reporting should measure what practitioners need, not what is convenient to compare. Through LLM-assisted, author-approved content analysis, we derive a 15-slot information schema from professional humanitarian situation reports using qualitative content analysis, then evaluate 121 human and 261 LLM-generated reports across five disasters and three hazard types. Under google/gemini-3.7-flash on Cyclone Idai, the best machine arm is nominally above the empirical human ceiling (0.827 vs. 0.821), but the difference is statistically indistinguishable from zero (95% bootstrap CI [-.049,.059]; permutation p=.852), and the ordering does not replicate under openai/gpt-5.6-luna. On a blind 100-item Idai sample, the author agrees with the Gemini judge at 0.910 accuracy and unweighted Cohen’s 𝜅=.799. All four judges clear the plan’s raw-agreement criterion of .7; only Gemini reaches .7 on chance-corrected 𝜅. Across the three primary events, machine reports reproduce 5.4–11.0% of same-day official figures. Across those same events, raw exact-number matching links 14.3–19.6% of official figures and 85.4–90.1% of machine figures to the social stream; manual audit confirms 45/50 declared absences, 4/25 official matches, and 24/25 machine matches semantically. The issuing-provenance gap is positive in all 12 observed event–judge cells: report date, organisation, series, and authority were not supplied at professional-report coverage in any cell. On the three primary events, ROUGE-L is uninformative, BERTScore is negatively associated with completeness after multiplicity correction, and a generic LLM judge changes sign.

Communication in times of crisis is essential. However, there is often a mismatch between the language of governments, aid providers, doctors, and those to whom they are providing aid. It is reasonable to turn to commercial MT tools in these scenarios. But how effective are these tools for translating to and from low resource languages, particularly in the crisis or medical domain? In this study, we evaluate 5 commercial systems used for MT (Google Translate, Microsoft Translator, GPT, Gemini and Claude) against the TICO-19 dataset, which contains pandemic-related sentences from a large set of high priority languages spoken by communities most likely to be adversely affected in the next pandemic. We assess the current degree of “readiness” for this next pandemic based on the usability of the output translations.

In this paper, we evaluate three state-of-the-art (SOTA) MM-LLMs on consistency of their outputs across text and audio modalities using real emergency alerts. Building on prior semantic-similarity and factual-overlap evaluation, we introduce a LIWC aligned analysis of modality dependent differences in linguistic structure, interpersonal framing, urgency, risk, and action-oriented language. Our findings indicate that despite being unified models, capable of taking in input from different modalities, no single model was able to achieve reliable cross-modal consistency. Persona-level results further indicate that cross-modal disparities vary across user groups. This introduces modality dependent inequity that unfairly affects individuals with access and functional needs. We release prompts, audio recordings, model outputs and evaluation material to support the crisis language technology community in building equitable solution for crisis management.

When a disaster strikes, question-answering (QA) systems promise to turn scattered satellite, infrastructure, and hazard impact data into actionable answers for coordinators working under time pressure. But how critically should we trust those answers, even if grounded in the background knowledge? We present RISE, a prototype knowledge-graph (KG)-powered QA system built on data from the June 2026 Venezuela M7.5 earthquake, and subject it to a critical evaluation. Two complementary studies run on the same 30 questions: a stage-wise trace records how each question moves through the pipeline, while an independent LLM-as-a-judge assessment scores the final answer on five language-quality criteria. Taken separately, both studies suggest a system that works: 87% intent accuracy, every question answered, and judge scores averaging above 4 out of 5. Albeit together, they reveal a different picture. Intent misclassifications trace to a linguistic ambiguity no keyword-based classifier can resolve: the same word serves as context in one question and as the information need in another, and lexical matching cannot tell them apart. High scores on grounding and appropriateness do not distinguish a correct answer from a fluent, well-cited answer to the wrong question, because the rubric scores the text against its evidence rather than against the question. Both findings share a common lesson for responsible AI in disaster management to design and develop effective support systems: fluent, well-grounded language by the AI agent is not evidence of a correct answer, and evaluating these systems on their text alone is not enough.

We describe our system for the LT4CPR 2026 shared task on automatic situation report generation, where a structured report must be produced from a single stream of crisis-related posts. Each stream is processed independently with one request to an open-weight large language model, gpt-oss-120b, whose zero-shot prompt restricts subsection labels to the official vocabulary; a deterministic step assembles the report schema. We designed the prompt from an error analysis of the 140 training streams, which showed that the untuned model’s reports were mostly correct but incomplete: they held about a third as many bullets as the references, cited about 1.7 posts per bullet against 4.4, and never produced several subsections. Asking instead for one consolidated bullet per storyline, citing every supporting post, raised the official primary score of our submission from 0.611 to 0.650 on the test set, third of nine teams. Replacing the model with nemotron-3-super under the original prompt lowered the score to 0.534, a larger effect than the prompt revision. We also report a negative result: giving the model a typed tracker of the figures stated in the posts did not increase the figures it reported, although the tracker held nearly twice as many reference figures as the model stated

Generating crisis situation reports from numerous short, redundant tweets requires both information organization and support from cited evidence. We present a system for the LT4CPR Shared Task that combines structured draft generation with bidirectional evidence-grounded refinement. Draft generation separates claim extraction, section-wise aggregation, report assembly, and summary generation. Refinement collects report-driven correction candidates and tweet-driven additions, verifies them against source tweets, and integrates accepted edits. Across all 23 cells covering two public development crises, our structured-generation configuration achieved a higher Main Score than direct generation (0.5308 vs. 0.4198). Adding bidirectional refinement slightly reduces Main Score to 0.5216 but increases the automatic NLI-based source-support metric for body bullets from 0.7925 to 0.8566. Our submission, combining both components, ranked first among the nine participating teams in the official test evaluation with a primary score of 0.682. A model-based audit of 50 randomly sampled modified body bullets judged 35 edits beneficial, eight equivalent, and seven detrimental, suggesting that bidirectional refinement improves statement quality in many edits. However, the audit and NLI disagree on individual improvements, suggesting that increases in the NLI metric should not be interpreted directly as equivalent gains in factual source support.

Automatic generation of crisis situation reports from social media requires transforming noisy, short-form text into structured, evidence-grounded prose, without access to information outside a single bounded time window. This paper describes an extractive pipeline submitted to a shared task on this problem: a supervised section classifier trained on tweet-to-bullet citation links, corroboration-based confidence estimation, safety-netted extractive bullet-text synthesis, and rule-based temporal filtering. The submitted system passes the task’s structural validator on all official test cells, spanning crisis types absent from training, and receives an official evaluation score of 0.480. Decomposing the two metrics composing this score shows that the system obtains a BERTScore F1 of 0.508 and a BLEURT score of 0.451, exposing a substantial difference between semantic similarity and learned fluency/adequacy evaluation. Development-time diagnostics, including a coreference-recall experiment and a per-section quality breakdown, provide independent evidence that extractive text generation is a major remaining limitation. Three architectural alternatives tested during development are reported with their measurements rather than omitted, and a scoped, safety-netted rewrite step is identified as a direct next architectural extension.

We present nemotron-120b-sitrep, an end-to-end four-stage generate-and-judge pipeline submitted to the First Shared Task on Automatic Situation Report Generation from Crisis Social Media Data at LT4CPR 2026 (AACL-IJCNLP 2026). Powered by NVIDIA’s Nemotron-3 Super 120B model hosted on the NVIDIA NIM API, the pipeline tackles the acute challenges of emergency reporting (social media noise, multi-section hierarchical taxonomies, and strict tweet-to-claim evidence attribution) by decomposing generation into: (1) rule-based ingestion and noise filtering, (2) unified template-filling synthesis across an 11-section situation report (SITREP) taxonomy under strict canonical subsection and evidence-grounding constraints, (3) cross-section deduplication, granular bullet consolidation, and dual-source confidence calibration via a Senior Judge LLM, and (4) Schema v1.2 JSON assembly. On the official shared task evaluation, our system achieves an Official Primary Score of 0.5733 on the 117-cell blind test set (BERTScore F1 of 0.5984 and BLEURT of 0.5483), demonstrating consistent performance that transfers to unseen crisis types, with scores of 0.5011 on the 140-cell training benchmark and 0.5057 on the 23-cell development benchmark. We provide a detailed calamity-by-calamity analysis across disaster scenarios, full algorithmic hyperparameters, and prompt engineering specifications.

Crisis situation report generation organizes evolving information into structured summaries with traceable supporting evidence. Generating these reports from noisy social-media streams requires balancing evidence completeness and concentration. Unfiltered evidence retains rare observations but includes competing material and remains subject to finite prompt budgets, while filtering and compression risk discarding useful sparse evidence. In this paper, we propose R01G-SR, a provenance-preserving approach that exploits complementary evidence views through whole-report selection. The approach generates two candidate reports using the same quantized 4B generator, presenting unfiltered chronological evidence to one route and filtered, near-duplicate-compressed evidence to the other. A logistic selector uses input- and report-derived features to choose one complete candidate without merging or rewriting reports, and a separate deterministic compliance layer enforces structural and evidence-ID requirements. Nested crisis-disjoint evaluation shows aggregate improvements over both fixed routes, and the official hidden-test evaluation shows improved report quality over the submitted compressed-route fallback. Analysis reveals larger selection gains on longer inputs and later windows, while early windows account for most remaining loss relative to selecting the better available candidate. These findings support evidence-view selection as a useful component of evidence-linked crisis reporting.

Structured crisis reports must preserve both information and its placement under canonical subsections. Merging reports generated from input chunks can omit subsections or duplicate their entries elsewhere. We introduce deterministic missing-subsection recovery and exact cross-title deduplication. The method reuses already generated model outputs, held fixed during postprocessing comparisons, without additional model calls. On 23 development cells, the combined postprocessor increased the official primary score from 0.4820 to 0.4890, a 1.45% relative gain over the baseline. Component comparisons attribute most of this development gain to recovery. On 117 test cells, all three submitted postprocessors scored 0.76–0.85% above the same baseline system on that split. Our primary system, GROUNDED, adds a fixed negative-template filter and placed fourth among nine teams; our highest-scoring submission also placed fourth in the best-system ranking. The results support structural correction of existing outputs, with small numerical differences between recovery policies across splits. They measure reference similarity, not factual entailment.

Language Technologies for Crisis Preparedness and Response (LT4CPR) encompass a broad range of natural language applications designed to support disaster response and preparedness. Within this domain, our shared task focuses specifically on leveraging social media data to extract actionable information and generate structured situation reports. To establish a formal benchmark, we introduce a domain-specific taxonomy for categorizing critical crisis information, alongside a novel pipeline for generating synthetic datasets to support task training and evaluation. Furthermore, we develop two baseline systems and propose an evaluation methodology designed to leverage the structural framework of situation reports. The shared task attracted participation from nine teams, whose diverse approaches offer valuable preliminary insights into the feasibility of structured situation report generation and the effectiveness of our evaluation setup. Finally, we highlight key open challenges (such as data access, operational alignment, noise filtering) which needs to be addressed in future research.