Mohammed Bennamoun

Author directory

2026

Video captioning directly reflects a Video Large Language Model’s perception, making caption fidelity a strong proxy for audio-visual grounding. Existing approaches often rely on costly annotations or stronger external teachers, limiting scalability. In this paper, we instead explore whether a VideoLLM can improve its own captions by exploiting structure already present within the model through self-supervision. Our exploration builds on two asymmetries we observe in current VideoLLMs. The first is a task asymmetry: captioning requires broad open-ended perception, whereas question answering narrows reasoning to specific dimensions. The second is an information asymmetry: the same model reasons more reliably with richer context or higher-fidelity audio-visual inputs. Motivated by these observations, we propose SODA (Self-Supervised On-Policy Distillation via Dual-Stream Auditing), a framework that enables a VideoLLM to critique and refine its own captions. A student first produces an initial caption from a low-budget video view, followed by a dual-stream audit: a Caption QA Stream identifies factual errors, and a Video QA Stream detects omitted content. Conditioned on this Revision QA Bank, a teacher using a larger input budget rewrites the caption, and the refined outputs are distilled back into the student. Experiments suggest that, despite training only on the captioning objective, SODA improves caption fidelity, reduces hallucination, and remains competitive on general video understanding.

2018

We present NovelPerspective: a tool to allow consumers to subset their digital literature, based on point of view (POV) character. Many novels have multiple main characters each with their own storyline running in parallel. A well-known example is George R. R. Martin’s novel: “A Game of Thrones”, and others from that series. Our tool detects the main character that each section is from the POV of, and allows the user to generate a new ebook with only those sections. This gives consumers new options in how they consume their media; allowing them to pursue the storylines sequentially, or skip chapters about characters they find boring. We present two heuristic-based baselines, and two machine learning based methods for the detection of the main character.
The evaluation of image caption quality is a challenging task, which requires the assessment of two main aspects in a caption: adequacy and fluency. These quality aspects can be judged using a combination of several linguistic features. However, most of the current image captioning metrics focus only on specific linguistic facets, such as the lexical or semantic, and fail to meet a satisfactory level of correlation with human judgements at the sentence-level. We propose a learning-based framework to incorporate the scores of a set of lexical and semantic metrics as features, to capture the adequacy and fluency of captions at different linguistic levels. Our experimental results demonstrate that composite metrics draw upon the strengths of stand-alone measures to yield improved correlation and accuracy.

2008