When a disaster strikes, question-answering (QA) systems promise to turn scattered satellite, infrastructure, and hazard impact data into actionable answers for coordinators working under time pressure. But how critically should we trust those answers, even if grounded in the background knowledge? We present RISE, a prototype knowledge-graph (KG)-powered QA system built on data from the June 2026 Venezuela M7.5 earthquake, and subject it to a critical evaluation. Two complementary studies run on the same 30 questions: a stage-wise trace records how each question moves through the pipeline, while an independent LLM-as-a-judge assessment scores the final answer on five language-quality criteria. Taken separately, both studies suggest a system that works: 87% intent accuracy, every question answered, and judge scores averaging above 4 out of 5. Albeit together, they reveal a different picture. Intent misclassifications trace to a linguistic ambiguity no keyword-based classifier can resolve: the same word serves as context in one question and as the information need in another, and lexical matching cannot tell them apart. High scores on grounding and appropriateness do not distinguish a correct answer from a fluent, well-cited answer to the wrong question, because the rubric scores the text against its evidence rather than against the question. Both findings share a common lesson for responsible AI in disaster management to design and develop effective support systems: fluent, well-grounded language by the AI agent is not evidence of a correct answer, and evaluating these systems on their text alone is not enough.