Kunhee Ryu
Author directory2026
Evaluating LLM Agents Beyond Local Edit Validity
Kunhee Ryu | Chi-Guhn Lee | Keeheon Lee
Findings of the Association for Computational Linguistics: EMNLP 2026
Kunhee Ryu | Chi-Guhn Lee | Keeheon Lee
Findings of the Association for Computational Linguistics: EMNLP 2026
Evaluations of sequential large language model (LLM) agents often report local action validity or final success. Delayed constructive tasks additionally require recoverability, meaning each accepted edit must leave the partial artifact completable within the remaining budget. We introduce a measurement protocol across Word Ladder, an alloy composition proxy, and the fitness landscape of the B1 domain of protein G (GB1). We call an episode surface clean when every action is well formed and locally admissible. Across six models, Word Ladder exposes interface failure with 36.1% local validity. Alloy and GB1 episodes are often surface clean, yet only 16.0% and 6.8% of them succeed. Exact enumeration of 625 GB1 variants over budgets from zero to six decomposes failure. Among surface clean episodes, 51.4% cross a recoverability boundary, while 14.1% of the remaining episodes succeed. Bounded composition audits and intervention checks connect bottlenecks to repair classes. These results establish recoverability as a decomposable and auditable evaluation axis that informs interventions for sequential agents.
Out-of-Distribution Checkpoint Selection Has a Resolution Problem: Auditing Sparse Exact Match with Token Likelihood
Kunhee Ryu | Keeheon Lee
Findings of the Association for Computational Linguistics: EMNLP 2026
Kunhee Ryu | Keeheon Lee
Findings of the Association for Computational Linguistics: EMNLP 2026
Exact match (EM) can make checkpoint selection ambiguous when a small labeled out of distribution (OOD) validation set leaves several checkpoints tied at zero or at the maximum. Without an explicit tie rule, the reported metric does not identify which checkpoint produced the final test score. We study this validation resolution problem in semantic parsing and morphological reinflection. Our protocol reports the reference budget, token likelihood choice under teacher forcing, and latest tied candidate, while using EM for a unique nonzero maximum. Across 600 draws from 30 runs, 8 references leave every early checkpoint at zero in 48% of draws and the maximum tied in 57%. Within tied draws, mean full split regret falls from 0.063 under random selection to 0.050 with token likelihood. The latest tied candidate performs better on average for mostly monotone trajectories, but token likelihood independently orders candidates that hard EM cannot distinguish. These results support reporting both orderings when small labeled OOD validation sets guide checkpoint selection.