Hyeji Choi
Author directory2026
When Accuracy Aligns and Fails: Diagnosing Human–LLM Response Alignment at Population Scale
Seoyoon Park | Hyeji Choi | Subin An | Xiaonan Wang | Heejae Kim | Joeun Kang | Hansaem Kim
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Seoyoon Park | Hyeji Choi | Subin An | Xiaonan Wang | Heejae Kim | Joeun Kang | Hansaem Kim
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
We propose CAPE, a Choice-Distribution Alignment and Performance Evaluation framework for diagnosing human–LLM response alignment using population-scale human choice distributions. CAPE separates outcome-level difficulty alignment from response-distribution alignment with four metrics: OA, MAE, PA, and DA. We apply CAPE to 609 Korean CSAT items with item-level response distributions from approximately 300K examinees per item, and evaluate 16 LLMs. Results show that model capability is the strongest predictor of alignment, with accuracy closely tied to both full-choice and distractor-only alignment. However, group-level scores are affected by aggregation size and model composition, and neither reasoning augmentation nor Korean adaptation yields consistent residual advantages. Bloom-level analyses further suggest that capability-linked alignment varies across item types. CAPE thus enables evaluation beyond accuracy by diagnosing how model success and failure patterns relate to human population responses.
2025
FLUID QA: A Multilingual Benchmark for Figurative Language Usage in Dialogue across English, Chinese, and Korean
Seoyoon Park | Hyeji Choi | Minseon Kim | Subin An | Xiaonan Wang | Gyuri Choi | Hansaem Kim
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Seoyoon Park | Hyeji Choi | Minseon Kim | Subin An | Xiaonan Wang | Gyuri Choi | Hansaem Kim
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Figurative language conveys stance, emotion, and social nuance, making its appropriate use essential in dialogue. While large language models (LLMs) often succeed in recognizing figurative expressions at the sentence level, their ability to use them coherently in conversation remains uncertain. We introduce FLUID QA, the first multilingual benchmark that evaluates figurative usage in dialogue across English, Korean, and Chinese. Each item embeds figurative choices into multi-turn contexts. To support interpretation, we include FLUTE-bi, a sentence-level diagnostic task. Results reveal a persistent gap: models that perform well on FLUTE-bi frequently fail on FLUID QA, especially in sarcasm and metaphor. These errors reflect systematic rhetorical confusion and limited discourse reasoning. FLUID QA provides a scalable framework for assessing usage-level figurative competence across languages.