Heesoo Kwon
Author directory2026
KoEVD: A Korean Benchmark for Metadata-Conditioned Multi-Stage LLM Safety Evaluation
San Kim | Wonjung Kim | Heesoo Kwon | Jae Hee Yang
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
San Kim | Wonjung Kim | Heesoo Kwon | Jae Hee Yang
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Large language model (LLM) safety benchmarks often evaluate input risk, response selection, generation, and tool use separately, making comparisons across tasks difficult. We introduce KoEVD (Korean Ethical Violation Dataset), which links five evaluation and analysis tasks through annotated Korean source utterances: input-risk judgment, candidate-response selection, generation safety, response-strategy analysis, and non-executable mock-tool selection. The tasks share source IDs, with task-specific subsets and candidate sampling where required. Built through an approximately 3.5-year human-governed, AI-assisted process, KoEVD contains 13,552 curated user utterances and 71,395 linked response candidates. Across 12 open-weight and commercial models, observed task profiles differ, and aggregate scores obscure variation across severity, explicitness, utterance type, and topic. In matched generation and mock-tool-selection cases, responses classified as safe coexist with unsafe mock-tool choices; pooled unsafe-call rates are 9.73% and 9.31% for the classifier-safe and classifier-unsafe groups, respectively, with framing-specific differences. These diagnostic measurements do not establish real-world safety. KoEVD supports detailed Korean safety analysis through contextual annotations and source-linked comparisons.