<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kim-2026-egovoice">
    <titleInfo>
        <title>EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Heeseung</namePart>
        <namePart type="family">Kim</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-459-0</identifier>
    </relatedItem>
    <abstract>Wearable augmented reality (AR) assistants are moving toward continuous real-world interaction, where they perceive the user’s activity through first-person video and audio and provide timely spoken guidance without being explicitly asked. While proactive video assistants, spoken dialog systems, and egocentric task understanding have each advanced rapidly, existing systems do not address the joint problem of deciding when to speak and what to say from continuous first-person streams. We introduce EgoVoice, a framework for training and evaluating proactive egocentric spoken assistants. From HoloAssist video recordings of real human instructors, we construct clean audio streams through source separation and speech resynthesis, and convert each video session into a format where the model must decide at each moment whether to remain silent or provide spoken guidance. We fine-tune an omni-modal LLM with our data, and further improve its proactive intervention behavior with direct preference optimization. Experiments across closed and open-source models show that existing systems rarely produce well-timed, meaningful proactive interventions, while EgoVoice yields clear improvements in intervention timing, content relevance, and human preference over the zero-shot backbone. We will release the dataset, code, and model checkpoints.</abstract>
    <identifier type="citekey">kim-2026-egovoice</identifier>
    <location>
        <url>https://aclanthology.org/2026.emnlp-main.1277/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>29112</start>
            <end>29136</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams
%A Kim, Heeseung
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F kim-2026-egovoice
%X Wearable augmented reality (AR) assistants are moving toward continuous real-world interaction, where they perceive the user’s activity through first-person video and audio and provide timely spoken guidance without being explicitly asked. While proactive video assistants, spoken dialog systems, and egocentric task understanding have each advanced rapidly, existing systems do not address the joint problem of deciding when to speak and what to say from continuous first-person streams. We introduce EgoVoice, a framework for training and evaluating proactive egocentric spoken assistants. From HoloAssist video recordings of real human instructors, we construct clean audio streams through source separation and speech resynthesis, and convert each video session into a format where the model must decide at each moment whether to remain silent or provide spoken guidance. We fine-tune an omni-modal LLM with our data, and further improve its proactive intervention behavior with direct preference optimization. Experiments across closed and open-source models show that existing systems rarely produce well-timed, meaningful proactive interventions, while EgoVoice yields clear improvements in intervention timing, content relevance, and human preference over the zero-shot backbone. We will release the dataset, code, and model checkpoints.
%U https://aclanthology.org/2026.emnlp-main.1277/
%P 29112-29136

Markdown (Informal)

[EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams](https://aclanthology.org/2026.emnlp-main.1277/) (Kim, EMNLP 2026)

• EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams (Kim, EMNLP 2026)

ACL

• Heeseung Kim. 2026. EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 29112–29136, Budapest, Hungary. Association for Computational Linguistics.