Yingzhe Fan

Author directory

2026

Language Technologies for Crisis Preparedness and Response (LT4CPR) encompass a broad range of natural language applications designed to support disaster response and preparedness. Within this domain, our shared task focuses specifically on leveraging social media data to extract actionable information and generate structured situation reports. To establish a formal benchmark, we introduce a domain-specific taxonomy for categorizing critical crisis information, alongside a novel pipeline for generating synthetic datasets to support task training and evaluation. Furthermore, we develop two baseline systems and propose an evaluation methodology designed to leverage the structural framework of situation reports. The shared task attracted participation from nine teams, whose diverse approaches offer valuable preliminary insights into the feasibility of structured situation report generation and the effectiveness of our evaluation setup. Finally, we highlight key open challenges (such as data access, operational alignment, noise filtering) which needs to be addressed in future research.
Automatic metrics stand in for native-speaker evaluation in low-resource retrieval-augmented generation (RAG), so their validity determines what can be concluded about these systems. We test whether common metrics detect re- trieval and grounding failure in one fixed RAG pipeline across ten Indic languages. They do not merely miss the failure in Santali; they invert it. Hit@5 is 0.011, yet raw XLM-R BERTScore is the highest in the set at 0.902, changes little when retrieved evidence is re- moved, and assigns nearly the same score to matched and unrelated references. The cause is coverage: the scorer maps 39% of Santali tokens to <unk>, and a scorer whose vocab- ulary includes Ol Chiki restores discrimina- tion while LaBSE does not. Retrieval fails for the same reason. The gold article is indexed for 80% of Santali queries, but the retriever shares XLM-R’s tokenizer and collapses before reranking. Judge-based faithfulness and con- text precision, computed by a model with a dif- ferent tokenizer, flag the failure independently. A metric that cannot represent a script does not become noisy; it becomes constant and high. We argue that metric validity in low-resource languages must be established per language, and that evidence removal, reference shuffling, and the scorer’s <unk> rate can be important additions.