Ye-Chan Kim
Author directory2026
Seeing Before Synthesizing: VLM-Guided Transition Event Discovery for Weakly-Supervised Dense Video Captioning
Ye-Chan Kim | Seunghee Choi | SeungJu Cha | Si-Woo Kim | Hwiseon Kim | HyunGee Kim | Dong-Jin Kim
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Ye-Chan Kim | Seunghee Choi | SeungJu Cha | Si-Woo Kim | Hwiseon Kim | HyunGee Kim | Dong-Jin Kim
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Weakly-Supervised Dense Video Captioning aims to localize and describe multiple events in untrimmed videos given only an ordered set of event-level captions per video. Recent work synthesizes auxiliary transition captions via LLM to provide additional vision-language alignment, but these captions lack visual grounding and are rigidly assigned to every inter-event gap at a fixed location and duration. To address these, we propose Seeing Before Synthesizing (SBS), a framework that adaptively provides visually grounded linguistic guidance only where warranted. Leveraging a VLM, we generate frame-level narratives for the inter-event gap and detect transitions from the semantic variation across them. For identified transitions, we then refine inter-event temporal masks by blending the temporal midpoint with the semantic change point and selecting the width that maximizes vision-language alignment. Experiments on ActivityNet Captions and YouCook2 demonstrate state-of-the-art performance in both captioning and localization.
2025
Sali4Vid: Saliency-Aware Video Reweighting and Adaptive Caption Retrieval for Dense Video Captioning
MinJu Jeon | Si-Woo Kim | Ye-Chan Kim | HyunGee Kim | Dong-Jin Kim
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
MinJu Jeon | Si-Woo Kim | Ye-Chan Kim | HyunGee Kim | Dong-Jin Kim
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Dense video captioning aims to temporally localize events in video and generate captions for each event. While recent works propose end-to-end models, they suffer from two limitations: (1) applying timestamp supervision only to text while treating all video frames equally, and (2) retrieving captions from fixed-size video chunks, overlooking scene transitions. To address these, we propose Sali4Vid, a simple yet effective saliency-aware framework. We introduce Saliency-aware Video Reweighting, which converts timestamp annotations into sigmoid-based frame importance weights, and Semantic-based Adaptive Caption Retrieval, which segments videos by frame similarity to capture scene transitions and improve caption retrieval. Sali4Vid achieves state-of-the-art results on YouCook2 and ViTT, demonstrating the benefit of jointly improving video weighting and retrieval for dense video captioning.