Large Language Models are increasingly optimized for deep reasoning, often through post-training objectives that strongly reward correct task completion. We ask whether this pressure toward correct completion can interfere with recognizing when a task should no longer be pursued. We introduce MortalMATH, a benchmark of 150 scenarios in which users request algebra help while describing increasingly urgent situations, from mild distraction to life-threatening emergencies. We find a striking contrast across models: several reasoning-oriented models, including Qwen3-32B and GPT-5 variants, continue solving the mathematical task with high accuracy even under extreme emergency contexts, while other models increasingly interrupt the task or acknowledge the danger. Reasoning models also spend substantial computation on the original problem, delaying safety-relevant information by several seconds. These results expose a failure mode of highly optimized task completion: training models to persist until they reach a verifiably correct answer may also make them less willing to stop when the surrounding context demands it.

This paper presents an overview of, and the results from, the 2025 Shared Task on Reproducibility of Evaluations in NLP (ReproNLP’25) which followed on from four previous shared tasks on reproducibility of evaluations, ReproNLP’24, ReproNLP’23, ReproGen’22 and ReproGen’21. This shared task series forms part of an ongoing research programme designed to develop theory and practice of reproducibility assessment in NLP and machine learning, against a backdrop of increasing recognition of the importance of the topic across the two fields. We describe the ReproNLP’25 shared task, summarise results from the reproduction studies submitted, and provide additional comparative analysis of their results, including for the first time additional, ‘sanity-check’ evaluations by LLMs.