Chanyoung Kim

Author directory

2026

Multimodal evaluation is strengthened when benchmarks reflect the languages, institutions, and disciplinary conventions that shape expert visual reasoning. We introduce KMMMU, a native Korean benchmark built from Korean-native examination materials. KMMMU contains 3,466 questions spanning nine disciplines and nine visual modality categories. For targeted evaluation on challenging questions and Korean-specific knowledge, we additionally provide KMMMU-Adv and KMMMU-Local, respectively. Experiments show that the strongest evaluated open-weight model reaches 42.05% on the KMMMU, while the best evaluated proprietary model achieves 52.42% on the KMMMU-Adv. Performance varies substantially across disciplines and answer formats, with Korean-specific questions showing gaps of up to 13.43%. Error analysis suggests that inspected failures often involve convention-to-label mapping, few-shot symbolic induction, localized knowledge recall, and domain-specific standards, not reasoning depth alone. KMMMU provides a testbed for native Korean expert-level multimodal understanding beyond English-centric or translation-based evaluation.
Vision–Language–Action (VLA) models achieve strong performance in robotic manipulation by leveraging pre-trained vision–language backbones. However, in downstream robotic settings, they are typically fine-tuned with limited data, leading to overfitting to specific instruction formulations and leaving robustness to paraphrased instructions underexplored. To study this gap, we introduce LIBERO-Para, a controlled benchmark that independently varies action expressions and object references for fine-grained analysis of linguistic generalization. Across seven VLA configurations (0.6B–7.5B), we observe consistent performance degradation of 22–52pp under paraphrasing. This degradation is primarily driven by object-level lexical variation: even simple synonym substitutions cause large drops, indicating reliance on surface-level matching rather than semantic grounding. Moreover, 80–96% of failures arise from planning-level trajectory divergence rather than execution errors, showing that paraphrasing disrupts task identification. Binary success rate treats all paraphrases equally, obscuring whether models perform consistently across difficulty levels or rely on easier cases. To address this, we propose PRIDE, a metric that quantifies paraphrase difficulty using semantic and syntactic factors. Our benchmark and code are available at https://github.com/cau-hai-lab/LIBERO-Para.