Hyunwoo Kim
Author directoryOther people with similar names: Hyunwoo Kim
Unverified author pages with similar names: Hyunwoo Kim
2026
XYBench: Can LLMs Respond Pragmatically to Queries with Misconceptions?
Akhila Yerukola | Jena D. Hwang | Mingqian Zheng | Jenna Godsey | Hyunwoo Kim | Valentina Pyatkin | Jennifer Hu | Maarten Sap
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Akhila Yerukola | Jena D. Hwang | Mingqian Zheng | Jenna Godsey | Hyunwoo Kim | Valentina Pyatkin | Jennifer Hu | Maarten Sap
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
When non-expert users ask LLMs for assistance, their queries can often have misconceptions (e.g., "How do I parse XML with regex?"). In such cases, often referred to as the XY-problem, LLMs must identify the misconception ("regex are fragile") and meaningfully direct the user toward a pragmatic solution that will address the root problem implicit in the request ("use an XML parser"). We introduce XYBench, a benchmark of 8,115 such queries, drawn from technical (StackOverflow/StackExchange) and everyday (WikiHow and a manually-curated subset) domains. We design an evaluation paradigm that assesses model responses along three criteria grounded in cooperative response theory: (a) presence and (b) emphasis on pragmatic solutions, and (c) identification of misconceptions. Our experiments show that even the strongest LLMs predominantly answer the literal request (0.75–0.92) and far less often the intended one (0.33–0.71), while substantially lagging behind humans at identifying misconceptions (at most 63% vs. 79–90%). Further, models overwhelmingly prefer pragmatic responses in a multiple choice setting yet consistently fail to generate them. Oracle ablation experiments show that providing explicit user intent at generation time helps; however a large gap remains, suggesting pragmatic redirection is a fundamentally underdeveloped capability in current LLMs.
2025
ALPACA AGAINST VICUNA: Using LLMs to Uncover Memorization of LLMs
Aly M. Kassem | Omar Mahmoud | Niloofar Mireshghallah | Hyunwoo Kim | Yulia Tsvetkov | Yejin Choi | Sherif Saad | Santu Rana
Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Aly M. Kassem | Omar Mahmoud | Niloofar Mireshghallah | Hyunwoo Kim | Yulia Tsvetkov | Yejin Choi | Sherif Saad | Santu Rana
Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
In this paper, we investigate the overlooked impact of instruction-tuning on memorization in large language models (LLMs), which has largely been studied in base, pre-trained models. We propose a black-box prompt optimization method where an attacker LLM agent uncovers higher levels of memorization in a victim agent, surpassing traditional approaches that prompt the model directly with training data. Using an iterative rejection-sampling process, we design instruction-based prompts that minimize overlap with training data to avoid providing direct solutions while maximizing overlap between the victim’s output and the training data to induce memorization. Our method shows 23.7% more overlap with training data compared to state-of-the-art baselines. We explore two attack settings: an analytical approach that determines the empirical upper bound of the attack, both with and without access to responses for prompt initialization, and a practical classifier-based method for assessing memorization without access to memorized data. Our findings reveal that instruction-tuned models can expose pre-training data as much as, or more than, base models; contexts beyond the original training data can lead to leakage; and instructions generated by other LLMs open new avenues for automated attacks, which we believe require further exploration.
Socratic-MCTS: Test-Time Visual Reasoning by Asking the Right Questions
David Acuna | Ximing Lu | Jaehun Jung | Hyunwoo Kim | Amlan Kar | Sanja Fidler | Yejin Choi
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
David Acuna | Ximing Lu | Jaehun Jung | Hyunwoo Kim | Amlan Kar | Sanja Fidler | Yejin Choi
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Recent research in vision-language models (VLMs) has centered around the possibility of equipping them with implicit long-form chain-of-thought reasoning—akin to the success observed in language models—via distillation and reinforcement learning. But what about the non-reasoning models already trained and deployed across the internet? Should we simply abandon them, or is there hope for a search mechanism that can elicit hidden knowledge and induce long reasoning traces— without any additional training or supervision? In this paper, we explore this possibility using a Monte Carlo Tree Search (MCTS)-inspired algorithm, which injects subquestion–subanswer pairs into the model’s output stream. We show that framing reasoning as a search process—where subquestions act as latent decisions within a broader inference trajectory—helps the model “connect the dots” between fragmented knowledge and produce extended reasoning traces in non-reasoning models. We evaluate our method across three benchmarks and observe consistent improvements. Notably, our approach yields a 2% overall improvement on MMMU-PRO, including a significant 9% gain in Liberal Arts.
2024
Deal, or no deal (or who knows)? Forecasting Uncertainty in Conversations using Large Language Models
Anthony Sicilia | Hyunwoo Kim | Khyathi Chandu | Malihe Alikhani | Jack Hessel
Findings of the Association for Computational Linguistics: ACL 2024
Anthony Sicilia | Hyunwoo Kim | Khyathi Chandu | Malihe Alikhani | Jack Hessel
Findings of the Association for Computational Linguistics: ACL 2024
Effective interlocutors account for the uncertain goals, beliefs, and emotions of others. But even the best human conversationalist cannot perfectly anticipate the trajectory of a dialogue. How well can language models represent inherent uncertainty in conversations? We propose FortUne Dial, an expansion of the long-standing “conversation forecasting” task: instead of just accuracy, evaluation is conducted with uncertainty-aware metrics, effectively enabling abstention on individual instances. We study two ways in which language models potentially represent outcome uncertainty (internally, using scores and directly, using tokens) and propose fine-tuning strategies to improve calibration of both representations. Experiments on eight difficult negotiation corpora demonstrate that our proposed fine-tuning strategies (a traditional supervision strategy and an off-policy reinforcement learning strategy) can calibrate smaller open-source models to compete with pre-trained models 10x their size.
Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs
Xuhui Zhou | Zhe Su | Tiwalayo Eisape | Hyunwoo Kim | Maarten Sap
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Xuhui Zhou | Zhe Su | Tiwalayo Eisape | Hyunwoo Kim | Maarten Sap
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Recent advances in large language models (LLM) have enabled richer social simulations, allowing for the study of various social phenomena. However, most recent work has used a more omniscient perspective on these simulations (e.g., single LLM to generate all interlocutors), which is fundamentally at odds with the non-omniscient, information asymmetric interactions that involve humans and AI agents in the real world. To examine these differences, we develop an evaluation framework to simulate social interactions with LLMs in various settings (omniscient, non-omniscient). Our experiments show that LLMs perform better in unrealistic, omniscient simulation settings but struggle in ones that more accurately reflect real-world conditions with information asymmetry. Moreover, we illustrate the limitations inherent in learning from omniscient simulations. Our findings indicate that addressing information asymmetry remains a fundamental challenge for LLM-based agents.
Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Models
Chani Jung | Dongkwan Kim | Jiho Jin | Jiseon Kim | Yeon Seonwoo | Yejin Choi | Alice Oh | Hyunwoo Kim
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Chani Jung | Dongkwan Kim | Jiho Jin | Jiseon Kim | Yeon Seonwoo | Yejin Choi | Alice Oh | Hyunwoo Kim
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
While humans naturally develop theory of mind (ToM), the capability to understand other people’s mental states and beliefs, state-of-the-art large language models (LLMs) underperform on simple ToM benchmarks. We posit that we can extend our understanding of LLMs’ ToM abilities by evaluating key human ToM precursors-perception inference and perception-to-belief inference-in LLMs. We introduce two datasets, Percept-ToMi and Percept-FANToM, to evaluate these precursory inferences for ToM in LLMs by annotating characters’ perceptions on ToMi and FANToM, respectively.Our evaluation of eight state-of-the-art LLMs reveals that the models generally perform well in perception inference while exhibiting limited capability in perception-to-belief inference (e.g., lack of inhibitory control).Based on these results, we present PercepToM, a novel ToM method leveraging LLMs’ strong perception inference capability while supplementing their limited perception-to-belief inference. Experimental results demonstrate that PercepToM significantly enhances LLM’s performance, especially in false belief scenarios.
Search
Co-authors
- Yejin Choi 3
- Maarten Sap 2
- David Acuna 1
- Malihe Alikhani 1
- Khyathi Chandu 1
- Tiwalayo Eisape 1
- Sanja Fidler 1
- Jenna Godsey 1
- Jack Hessel 1
- Jennifer Hu 1
- Jena D. Hwang 1
- Jiho Jin 1
- Chani Jung 1
- Jaehun Jung 1
- Amlan Kar 1
- Aly M. Kassem 1
- Dongkwan Kim 1
- Jiseon Kim 1
- Ximing Lu 1
- Omar Mahmoud 1
- Niloofar Mireshghallah 1
- Alice Oh 1
- Valentina Pyatkin 1
- Santu Rana 1
- Sherif Saad 1
- Yeon Seonwoo 1
- Anthony Sicilia 1
- Zhe Su 1
- Yulia Tsvetkov 1
- Akhila Yerukola 1
- Mingqian Zheng 1
- Xuhui Zhou 1