Jihwan Kim

Author directory

2026

Recent progress in Omni-models has yielded strong performance on benchmarks focused on static observations. However, their ability to maintain spatiotemporal continuity when predicting coherent future states from previous context remains underexplored. We address this by formulating Next Scene Prediction (NSP) as a diagnostic task for assessing multimodal near-future state prediction, requiring models to anticipate the immediate future from a given context. We introduce OmniNSP, a benchmark evaluating causal and cross-modal coherence across physical, logical, and social dynamics. Unlike prior tasks that often rely on textual or discriminative proxies, OmniNSP asks models to express predicted future states through text-mediated visual and speech synthesis. Our analysis shows that current Omni-models can produce plausible textual forecasts while still failing to preserve future-state consistency in generated outputs. A frozen-generator control substantially narrows cross-model visual-consistency gaps, attributing much of the original system-level difference to model-specific realization stacks rather than textual foresight alone. A modality-input ablation further shows that audio provides complementary predictive cues beyond vision, with the largest gains for logical transitions and contexts with informative speech. These results position OmniNSP as a diagnostic of both near-future prediction and the preservation of predicted state through multimodal realization.
Retrieving relevant tables from extensive databases for a given natural language query is essential for accurately answering questions in tasks such as text-to-SQL. Existing table retrieval approaches select a pre-determined set of k tables with the highest similarity to the query. However, the number of required tables varies across queries and cannot be known in advance. Enforcing a fixed number of retrieved tables regardless of the query may either retrieve an undersized set, failing to obtain all necessary evidence, or retrieve an oversized pool, including irrelevant tables. To address this issue, we propose an adaptive table retrieval method that adjusts the number of tables retrieved according to the requirements of each query. Specifically, we utilize an adaptive thresholding mechanism to selectively retrieve tables and integrate a sliding-window reranking algorithm to efficiently process a large table corpus. Extensive experiments on Spider, BIRD, and Spider 2.0 demonstrate that our method effectively addresses the limitations of the top-k retrieval strategy, improving performance in retrieval and downstream tasks. Our code and data are available at https://anonymous.4open.science/r/Adaptive-Table-Retrieval.
Reinforcement Learning with Verifiable Rewards (RLVR) can improve reasoning in large language models, but specialized non-English domains expose a data mismatch. Raw English domain text is often available, while target-language RLVR tasks with verifiable rewards are scarce. We introduce code-switched task synthesis for cross-lingual RLVR, a data-centric approach that derives target-language and code-switched RLVR task variants from otherwise unverifiable English domain text. Using fill-in-the-middle multiple-choice QA (FIM-MCQA) as a seed task format, we synthesize automatically verifiable tasks from domain passages and reshape them into cross-lingual variants. These variants preserve the same evidence-answer relation but change how English and target-language evidence are presented during RLVR. On Korean medical benchmarks and cybersecurity benchmarks across four target languages, code-switched training outperforms matched monolingual and bilingual-mixture baselines on average target-language performance while maintaining comparable English performance. Additional analyses provide evidence of better target-language input understanding and stronger cross-lingual alignment in both model outputs and internal representations. These results suggest that code-switched training helps target-language inputs access domain reasoning learned more strongly in English.
As the paradigm of AI shifts from text-based LLMs to Speech Language Models (SLMs), there is a growing demand for full-duplex systems capable of real-time, natural human-computer interaction.However, the development of such models is constrained by the scarcity of high-quality, multi-speaker conversational data, as existing large-scale resources are predominantly single-speaker or limited in volume.Addressing the complex dynamics of natural dialogue, such as overlapping and back-channeling remains a challenge, with standard processing pipelines suffering from diarization errors and ASR hallucinations.To bridge this gap, we present a robust and scalable open-source data processing pipeline designed for full-duplex model.Our code and project page are publicly available at https://anonymous-2001-j.github.io/sommelier.github.io/.