<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="li-lukasiewicz-2022-learning">
    <titleInfo>
        <title>Learning to Model Multimodal Semantic Alignment for Story Visualization</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Bowen</namePart>
        <namePart type="family">Li</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Thomas</namePart>
        <namePart type="family">Lukasiewicz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2022</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yoav</namePart>
            <namePart type="family">Goldberg</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Zornitsa</namePart>
            <namePart type="family">Kozareva</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yue</namePart>
            <namePart type="family">Zhang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Abu Dhabi, United Arab Emirates</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Story visualization aims to generate a sequence of images to narrate each sentence in a multi-sentence story, where the images should be realistic and keep global consistency across dynamic scenes and characters. Current works face the problem of semantic misalignment because of their fixed architecture and diversity of input modalities. To address this problem, we explore the semantic alignment between text and image representations by learning to match their semantic levels in the GAN-based generative model. More specifically, we introduce dynamic interactions according to learning to dynamically explore various semantic depths and fuse the different-modal information at a matched semantic level, which thus relieves the text-image semantic misalignment problem. Extensive experiments on different datasets demonstrate the improvements of our approach, neither using segmentation masks nor auxiliary captioning networks, on image quality and story consistency, compared with state-of-the-art methods.</abstract>
    <identifier type="citekey">li-lukasiewicz-2022-learning</identifier>
    <identifier type="doi">10.18653/v1/2022.findings-emnlp.346</identifier>
    <location>
        <url>https://aclanthology.org/2022.findings-emnlp.346</url>
    </location>
    <part>
        <date>2022-12</date>
        <extent unit="page">
            <start>4712</start>
            <end>4718</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Learning to Model Multimodal Semantic Alignment for Story Visualization
%A Li, Bowen
%A Lukasiewicz, Thomas
%Y Goldberg, Yoav
%Y Kozareva, Zornitsa
%Y Zhang, Yue
%S Findings of the Association for Computational Linguistics: EMNLP 2022
%D 2022
%8 December
%I Association for Computational Linguistics
%C Abu Dhabi, United Arab Emirates
%F li-lukasiewicz-2022-learning
%X Story visualization aims to generate a sequence of images to narrate each sentence in a multi-sentence story, where the images should be realistic and keep global consistency across dynamic scenes and characters. Current works face the problem of semantic misalignment because of their fixed architecture and diversity of input modalities. To address this problem, we explore the semantic alignment between text and image representations by learning to match their semantic levels in the GAN-based generative model. More specifically, we introduce dynamic interactions according to learning to dynamically explore various semantic depths and fuse the different-modal information at a matched semantic level, which thus relieves the text-image semantic misalignment problem. Extensive experiments on different datasets demonstrate the improvements of our approach, neither using segmentation masks nor auxiliary captioning networks, on image quality and story consistency, compared with state-of-the-art methods.
%R 10.18653/v1/2022.findings-emnlp.346
%U https://aclanthology.org/2022.findings-emnlp.346
%U https://doi.org/10.18653/v1/2022.findings-emnlp.346
%P 4712-4718

Markdown (Informal)

[Learning to Model Multimodal Semantic Alignment for Story Visualization](https://aclanthology.org/2022.findings-emnlp.346) (Li & Lukasiewicz, Findings 2022)

• Learning to Model Multimodal Semantic Alignment for Story Visualization (Li & Lukasiewicz, Findings 2022)

ACL

• Bowen Li and Thomas Lukasiewicz. 2022. Learning to Model Multimodal Semantic Alignment for Story Visualization. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4712–4718, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.