<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="chang-li-2026-causal">
    <titleInfo>
        <title>Causal Head Gating: Separating Visual Evidence and Knowledge Priors for Hallucination Mitigation in Multimodal LLMs</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Jiale</namePart>
        <namePart type="family">Chang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ying</namePart>
        <namePart type="family">Li</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2026</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-462-0</identifier>
    </relatedItem>
    <abstract>Multimodal large language models excel at vision-language tasks but suffer from hallucinations that undermine their reliability. Existing mitigation methods apply uniform heuristics at the output or attention level, overlooking the heterogeneous causal roles of attention heads and their dynamic behaviors during inference. We revisit hallucination from a causal perspective and identify it as competition between two pathways: a direct path from visual evidence to outputs, and an indirect path dominated by knowledge priors. When visual evidence weakens, MLLMs increasingly rely on parametric knowledge priors—such as spurious correlations learned during pretraining. This over-reliance causes attention heads responsible for visual grounding to drift toward language-driven patterns, resulting in generations consistent with the language model but unfaithful to the image. We propose Causal Head Gating, an inference-time framework that separates visual evidence from knowledge priors by modeling their causal contributions at the attention-head level. Under this framework, we introduce CHERI (Causal Head Evidence Reweighting and Intervention), which identifies attention heads whose visual contributions weaken under hallucination and dynamically restores their influence. Specifically, CHERI reinforces heads that propagate visual evidence while suppressing those dominated by knowledge priors, strengthening the direct causal effect of vision on generation and attenuating spurious indirect effects. Experiments across multiple benchmarks show that CHERI consistently reduces hallucination rates across diverse task formats while preserving overall generation quality and task performance.</abstract>
    <identifier type="citekey">chang-li-2026-causal</identifier>
    <location>
        <url>https://aclanthology.org/2026.findings-emnlp.511/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>10700</start>
            <end>10714</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Causal Head Gating: Separating Visual Evidence and Knowledge Priors for Hallucination Mitigation in Multimodal LLMs
%A Chang, Jiale
%A Li, Ying
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Findings of the Association for Computational Linguistics: EMNLP 2026
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-462-0
%F chang-li-2026-causal
%X Multimodal large language models excel at vision-language tasks but suffer from hallucinations that undermine their reliability. Existing mitigation methods apply uniform heuristics at the output or attention level, overlooking the heterogeneous causal roles of attention heads and their dynamic behaviors during inference. We revisit hallucination from a causal perspective and identify it as competition between two pathways: a direct path from visual evidence to outputs, and an indirect path dominated by knowledge priors. When visual evidence weakens, MLLMs increasingly rely on parametric knowledge priors—such as spurious correlations learned during pretraining. This over-reliance causes attention heads responsible for visual grounding to drift toward language-driven patterns, resulting in generations consistent with the language model but unfaithful to the image. We propose Causal Head Gating, an inference-time framework that separates visual evidence from knowledge priors by modeling their causal contributions at the attention-head level. Under this framework, we introduce CHERI (Causal Head Evidence Reweighting and Intervention), which identifies attention heads whose visual contributions weaken under hallucination and dynamically restores their influence. Specifically, CHERI reinforces heads that propagate visual evidence while suppressing those dominated by knowledge priors, strengthening the direct causal effect of vision on generation and attenuating spurious indirect effects. Experiments across multiple benchmarks show that CHERI consistently reduces hallucination rates across diverse task formats while preserving overall generation quality and task performance.
%U https://aclanthology.org/2026.findings-emnlp.511/
%P 10700-10714

Markdown (Informal)

[Causal Head Gating: Separating Visual Evidence and Knowledge Priors for Hallucination Mitigation in Multimodal LLMs](https://aclanthology.org/2026.findings-emnlp.511/) (Chang & Li, Findings 2026)

• Causal Head Gating: Separating Visual Evidence and Knowledge Priors for Hallucination Mitigation in Multimodal LLMs (Chang & Li, Findings 2026)

ACL

• Jiale Chang and Ying Li. 2026. Causal Head Gating: Separating Visual Evidence and Knowledge Priors for Hallucination Mitigation in Multimodal LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 10700–10714, Budapest, Hungary. Association for Computational Linguistics.