Jonghyun Lee
Author directory2026
MathQueue: Reasoning Scheduling Helps Only Within a Narrow Budget, and Answer Extraction Decides Whether It Helps at All
Jonghyun Lee | Byungjun Kim
The Proceedings of the 4th Workshop on Mathematical Natural Language Processing (MathNLP 2026)
Jonghyun Lee | Byungjun Kim
The Proceedings of the 4th Workshop on Mathematical Natural Language Processing (MathNLP 2026)
Chain-of-Thought (CoT) prompting treats every reasoning step as equally important, even though solving a mathematical problem depends largely on deciding which step to take first. We therefore formulate reasoning as a scheduling problem. MathQueue decomposes a problem into tasks, scores each task by how much subsequent work it enables, how much information it yields, and how likely it is to introduce an error, and then solves the tasks in that order. We evaluate this formulation on GSM8K with two models under a fixed limit on the number of tokens the model may generate, and find that it helps over a far narrower range of conditions than anticipated.
Executing each task in a separate model call, with a verifier that may trigger replanning, performs substantially worse than standard CoT (83% to 72%) at three times the generated tokens and, once the re-sent context of twelve calls is counted, thirteen times the total; the replanning loop accounts for most of the loss. A single-pass variant performs better, but for reasons unrelated to prioritisation: it terminates in the required answer format on every problem, whereas CoT does so on only 87%, and it generates a third fewer tokens with a quarter of the variance in length, yet conditional on an answer being delivered it is never more accurate than CoT.
Retaining this structure but suppressing the ranking leaves accuracy unchanged at loose budgets and costs 19 points at the single budget tight enough to make ordering consequential. Intervening on the scheduler directly rather than on the prompt then locates that effect: the task graphs the model generates are near-chains, so our priority rule reorders only 9 of 100 problems, and the gain therefore comes from committing to a ranked plan rather than from the particular scores we assign to tasks.
Whether any of this constitutes an improvement depends on a factor we had not expected to matter: how the answer is parsed from a truncated generation. Scoring only well-formed answers favours MathQueue in all eight comparisons; reading the last number instead reverses six of them. We derive an identity explaining the reversal, adjudicate between the rules by manually inspecting every non-compliant generation, and confirm on a second model that what STRICT scoring rewards is the baseline’s format compliance rather than its reasoning. Budget-constrained evaluation of mathematical reasoning must therefore fix and report how the answer is read.
Executing each task in a separate model call, with a verifier that may trigger replanning, performs substantially worse than standard CoT (83% to 72%) at three times the generated tokens and, once the re-sent context of twelve calls is counted, thirteen times the total; the replanning loop accounts for most of the loss. A single-pass variant performs better, but for reasons unrelated to prioritisation: it terminates in the required answer format on every problem, whereas CoT does so on only 87%, and it generates a third fewer tokens with a quarter of the variance in length, yet conditional on an answer being delivered it is never more accurate than CoT.
Retaining this structure but suppressing the ranking leaves accuracy unchanged at loose budgets and costs 19 points at the single budget tight enough to make ordering consequential. Intervening on the scheduler directly rather than on the prompt then locates that effect: the task graphs the model generates are near-chains, so our priority rule reorders only 9 of 100 problems, and the gain therefore comes from committing to a ranked plan rather than from the particular scores we assign to tasks.
Whether any of this constitutes an improvement depends on a factor we had not expected to matter: how the answer is parsed from a truncated generation. Scoring only well-formed answers favours MathQueue in all eight comparisons; reading the last number instead reverses six of them. We derive an identity explaining the reversal, adjudicate between the rules by manually inspecting every non-compliant generation, and confirm on a second model that what STRICT scoring rewards is the baseline’s format compliance rather than its reasoning. Budget-constrained evaluation of mathematical reasoning must therefore fix and report how the answer is read.
KVoiceBench, KOpenAudioBench, and KMMAU: Agent-Driven Korean Speech Benchmarks for Evaluating SpeechLMs
Haechan Kim | Seungjun Chung | Inkyu Park | Jihoo Lee | Jonghyun Lee
Findings of the Association for Computational Linguistics: EMNLP 2026
Haechan Kim | Seungjun Chung | Inkyu Park | Jihoo Lee | Jonghyun Lee
Findings of the Association for Computational Linguistics: EMNLP 2026
Speech language models (SpeechLMs) have achieved substantial progress by extending large language models (LLMs) to the speech modality. However, SpeechLM evaluation remains heavily centered on English, limiting reliable assessment of multilingual speech capabilities. Straightforward benchmark transfer through ASR, translation, normalization, and TTS can corrupt language-specific instructions, answer constraints, and spoken forms; for audio understanding, transferring source-language audio also fails to preserve target-language speaker attributes, accents, and paralinguistic properties. To address these limitations, we propose two human-agent benchmark-construction frameworks: one transfers source-language SpokenQA benchmarks into target-language SpokenQA benchmarks, and the other converts target-language ASR corpora into audio understanding benchmarks using transcriptions and speaker metadata. Using these frameworks, we construct and publicly release three Korean speech benchmarks: KVoiceBench and KOpenAudioBench for Korean SpokenQA, and KMMAU for Korean audio understanding, comprising 12,345 samples in total. We evaluate eight recent SpeechLMs and find that English–Korean performance gaps vary substantially across models and task families, and that SpokenQA and audio understanding rankings diverge, revealing complementary weaknesses invisible to English-only evaluation.