Multi-modal, Multi-Speaker Dialogue Turns Modeling with Multi-modal LLMs

Lukas Hilgert; Jan Niehues

Multi-modal, Multi-Speaker Dialogue Turns Modeling with Multi-modal LLMs

Correct Metadata for

Use this form to create a GitHub issue with structured data describing the correction. You will need a GitHub account. Once you create that issue, the correction will be reviewed by a staff member.

⚠️ Mobile Users: Submitting this form to create a new issue will only work with github.com, not the GitHub Mobile app.

Important: The Anthology treat PDFs as authoritative. Please use this form only to correct data that is out of line with the PDF. See our corrections guidelines if you need to change the PDF.

Title Adjust the title. Retain tags such as <fixed-case>.

Authors Adjust author names and order to match the PDF.

Abstract Correct abstract if needed. Retain XML formatting tags such as <tex-math>. You may use ... for bold, ... for italic, ... for underline, <sc>...</sc> for small-caps, <tt>...<tt> for typewriter text, <url>...</url> for URLs, <a href=...> for hyperlinks, and <par/> for paragraph breaks.

Verification against PDF Ensure that the new title/authors match the snapshot below. (If there is no snapshot or it is too small, consult the PDF.)

Authors concatenated from the text boxes above:

ALL author names match the snapshot above—including middle initials, hyphens, and accents.

Abstract

State-of-the-art dialogue systems focus on two-person dialogue and explicit indication of who is speaking. However, for using LLM-based AI systems in multi-speaker environments, it is necessary to determine who is speaking next. In this work, we investigate how the task can be formulated and how multi-modal LLMs are able to perform it. First, we frame the task as multi-modal Next Speaker Prediction (mNSP) and create a multi-modal, multi-speaker NSP dataset from multiple existing dialogue datasets with different task setups. Second, we build dedicated systems for the mNSP task using multi-modal LLMs, and finally perform automatic evaluation, comparing them to rule-based baselines.

Anthology ID:: 2026.mint-1.1
Volume:: Proceedings of the 1st Workshop on Multimodal Interaction in Face-to-Face Dialogue
Month:: October
Year:: 2026
Address:: Budapest, Hungary
Editors:: Esam Ghaleb, Ece Takmaz, Diego Frassinelli, Bulat Khaertdinov, Raquel Fernández, Zerrin Yumak, Asli Ozyurek
Venues:: MINT | WS
SIG:
Publisher:: Association for Computational Linguistics
Note:
Pages:: 1–9
Language:
URL:: https://aclanthology.org/2026.mint-1.1/
DOI:
Bibkey:
Cite (ACL):: Lukas Hilgert and Jan Niehues. 2026. Multi-modal, Multi-Speaker Dialogue Turns Modeling with Multi-modal LLMs. In Proceedings of the 1st Workshop on Multimodal Interaction in Face-to-Face Dialogue, pages 1–9, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):: Multi-modal, Multi-Speaker Dialogue Turns Modeling with Multi-modal LLMs (Hilgert & Niehues, MINT 2026)
Copy Citation:
PDF:: https://aclanthology.org/2026.mint-1.1.pdf

PDF Cite Search Fix data

Export citation

BibTeX
MODS XML
Endnote
Preformatted

@inproceedings{hilgert-niehues-2026-multi,
    title = "Multi-modal, Multi-Speaker Dialogue Turns Modeling with Multi-modal {LLM}s",
    author = "Hilgert, Lukas  and
      Niehues, Jan",
    editor = "Ghaleb, Esam  and
      Takmaz, Ece  and
      Frassinelli, Diego  and
      Khaertdinov, Bulat  and
      Fern{\'a}ndez, Raquel  and
      Yumak, Zerrin  and
      Ozyurek, Asli",
    booktitle = "Proceedings of the 1st Workshop on Multimodal Interaction in Face-to-Face Dialogue",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.mint-1.1/",
    pages = "1--9",
    ISBN = "979-8-89176-482-8",
    abstract = "State-of-the-art dialogue systems focus on two-person dialogue and explicit indication of who is speaking. However, for using LLM-based AI systems in multi-speaker environments, it is necessary to determine who is speaking next. In this work, we investigate how the task can be formulated and how multi-modal LLMs are able to perform it. First, we frame the task as multi-modal Next Speaker Prediction (mNSP) and create a multi-modal, multi-speaker NSP dataset from multiple existing dialogue datasets with different task setups. Second, we build dedicated systems for the mNSP task using multi-modal LLMs, and finally perform automatic evaluation, comparing them to rule-based baselines."
}

Download as File

<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="hilgert-niehues-2026-multi">
    <titleInfo>
        <title>Multi-modal, Multi-Speaker Dialogue Turns Modeling with Multi-modal LLMs</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Lukas</namePart>
        <namePart type="family">Hilgert</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jan</namePart>
        <namePart type="family">Niehues</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 1st Workshop on Multimodal Interaction in Face-to-Face Dialogue</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Esam</namePart>
            <namePart type="family">Ghaleb</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ece</namePart>
            <namePart type="family">Takmaz</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Diego</namePart>
            <namePart type="family">Frassinelli</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Bulat</namePart>
            <namePart type="family">Khaertdinov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Raquel</namePart>
            <namePart type="family">Fernández</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Zerrin</namePart>
            <namePart type="family">Yumak</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Asli</namePart>
            <namePart type="family">Ozyurek</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-482-8</identifier>
    </relatedItem>
    <abstract>State-of-the-art dialogue systems focus on two-person dialogue and explicit indication of who is speaking. However, for using LLM-based AI systems in multi-speaker environments, it is necessary to determine who is speaking next. In this work, we investigate how the task can be formulated and how multi-modal LLMs are able to perform it. First, we frame the task as multi-modal Next Speaker Prediction (mNSP) and create a multi-modal, multi-speaker NSP dataset from multiple existing dialogue datasets with different task setups. Second, we build dedicated systems for the mNSP task using multi-modal LLMs, and finally perform automatic evaluation, comparing them to rule-based baselines.</abstract>
    <identifier type="citekey">hilgert-niehues-2026-multi</identifier>
    <location>
        <url>https://aclanthology.org/2026.mint-1.1/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>1</start>
            <end>9</end>
        </extent>
    </part>
</mods>
</modsCollection>

Download as File

%0 Conference Proceedings
%T Multi-modal, Multi-Speaker Dialogue Turns Modeling with Multi-modal LLMs
%A Hilgert, Lukas
%A Niehues, Jan
%Y Ghaleb, Esam
%Y Takmaz, Ece
%Y Frassinelli, Diego
%Y Khaertdinov, Bulat
%Y Fernández, Raquel
%Y Yumak, Zerrin
%Y Ozyurek, Asli
%S Proceedings of the 1st Workshop on Multimodal Interaction in Face-to-Face Dialogue
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-482-8
%F hilgert-niehues-2026-multi
%X State-of-the-art dialogue systems focus on two-person dialogue and explicit indication of who is speaking. However, for using LLM-based AI systems in multi-speaker environments, it is necessary to determine who is speaking next. In this work, we investigate how the task can be formulated and how multi-modal LLMs are able to perform it. First, we frame the task as multi-modal Next Speaker Prediction (mNSP) and create a multi-modal, multi-speaker NSP dataset from multiple existing dialogue datasets with different task setups. Second, we build dedicated systems for the mNSP task using multi-modal LLMs, and finally perform automatic evaluation, comparing them to rule-based baselines.
%U https://aclanthology.org/2026.mint-1.1/
%P 1-9

Download as File

Markdown (Informal)

[Multi-modal, Multi-Speaker Dialogue Turns Modeling with Multi-modal LLMs](https://aclanthology.org/2026.mint-1.1/) (Hilgert & Niehues, MINT 2026)

Multi-modal, Multi-Speaker Dialogue Turns Modeling with Multi-modal LLMs (Hilgert & Niehues, MINT 2026)

ACL

Lukas Hilgert and Jan Niehues. 2026. Multi-modal, Multi-Speaker Dialogue Turns Modeling with Multi-modal LLMs. In Proceedings of the 1st Workshop on Multimodal Interaction in Face-to-Face Dialogue, pages 1–9, Budapest, Hungary. Association for Computational Linguistics.