Safety evaluations of large language models (LLMs) have largely focused on whether models refuse explicitly harmful requests. However, in high-stakes decision-support settings, seemingly legitimate requests may conceal missing information or hidden risk signals. We propose LogicLure, a benchmark for evaluating whether LLMs avoid premature decisions and actively seek evidence under such plausible scenarios with hidden risks. The benchmark spans eight domains and pairs persuasive visible narratives with hidden evidence maps accessible only through verification. Our evaluation measures not only the decision of models, but also whether they ask verification questions and whether their final decision is grounded in the revealed evidence. In our evaluation, the strongest model reaches only 26.4% evidence-grounded refusal despite 94.7% overall refusal. Furthermore, we analyze how scenario surface cues, explicit scrutiny setting, and prompt-induced stance shape these decisions. Together, these analyses characterize the key behavioral factors underlying LLM performance on LogicLure. The benchmark and code will be made publicly available.

Text classification with extremely weak supervision (EWS) imposes stricter supervision constraints compared to regular weakly supervise classification. Absolutely no labeled training samples or hand-crafted rules specific to the evaluation data are allowed. Such restrictions limit state-of-the-art EWS classification methods to indirect weak labeling techniques that assign unnatural label uncertainty estimates. We present PLAT, a framework that creates weak labels by leveraging recent developments in zero-shot text classification. PLAT employs models trained for sub-tasks other than classification to label documents. Most importantly, PLAT refrains from assigning overly confident weak labels and improves soft-label training performance for downstream classifiers. Classifiers trained with PLAT significantly outperform those trained on weak labels generated by the previous state-of-the-art in extremely weakly supervised text classification.