<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ilinykh-etal-2022-examining">
    <titleInfo>
        <title>Examining the Effects of Language-and-Vision Data Augmentation for Generation of Descriptions of Human Faces</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Nikolai</namePart>
        <namePart type="family">Ilinykh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Rafal</namePart>
        <namePart type="family">Černiavski</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Eva</namePart>
        <namePart type="given">Elžbieta</namePart>
        <namePart type="family">Sventickaitė</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Viktorija</namePart>
        <namePart type="family">Buzaitė</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Simon</namePart>
        <namePart type="family">Dobnik</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-06</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2nd Workshop on People in Vision, Language, and the Mind</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Patrizia</namePart>
            <namePart type="family">Paggio</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Albert</namePart>
            <namePart type="family">Gatt</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Marc</namePart>
            <namePart type="family">Tanti</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>European Language Resources Association</publisher>
            <place>
                <placeTerm type="text">Marseille, France</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>We investigate how different augmentation techniques on both textual and visual representations affect the performance of the face description generation model. Specifically, we provide the model with either original images, sketches of faces, facial composites or distorted images. In addition, on the language side, we experiment with different methods to augment the original dataset with paraphrased captions, which are semantically equivalent to the original ones, but differ in terms of their form. We also examine if augmenting the dataset with descriptions from a different domain (e.g., image captions of real-world images) has an effect on the performance of the models. We train models on different combinations of visual and linguistic features and perform both (i) automatic evaluation of generated captions and (ii) examination of how useful different visual features are for the task of facial feature classification. Our results show that although original images encode the best possible representation for the task, the model trained on sketches can still perform relatively well. We also observe that augmenting the dataset with descriptions from a different domain can boost performance of the model. We conclude that face description generation systems are more susceptible to language rather than vision data augmentation. Overall, we demonstrate that face caption generation models display a strong imbalance in the utilisation of language and vision modalities, indicating a lack of proper information fusion. We also describe ethical implications of our study and argue that future work on human face description generation should create better, more representative datasets.</abstract>
    <identifier type="citekey">ilinykh-etal-2022-examining</identifier>
    <location>
        <url>https://aclanthology.org/2022.pvlam-1.5</url>
    </location>
    <part>
        <date>2022-06</date>
        <extent unit="page">
            <start>26</start>
            <end>40</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Examining the Effects of Language-and-Vision Data Augmentation for Generation of Descriptions of Human Faces
%A Ilinykh, Nikolai
%A Černiavski, Rafal
%A Sventickaitė, Eva Elžbieta
%A Buzaitė, Viktorija
%A Dobnik, Simon
%Y Paggio, Patrizia
%Y Gatt, Albert
%Y Tanti, Marc
%S Proceedings of the 2nd Workshop on People in Vision, Language, and the Mind
%D 2022
%8 June
%I European Language Resources Association
%C Marseille, France
%F ilinykh-etal-2022-examining
%X We investigate how different augmentation techniques on both textual and visual representations affect the performance of the face description generation model. Specifically, we provide the model with either original images, sketches of faces, facial composites or distorted images. In addition, on the language side, we experiment with different methods to augment the original dataset with paraphrased captions, which are semantically equivalent to the original ones, but differ in terms of their form. We also examine if augmenting the dataset with descriptions from a different domain (e.g., image captions of real-world images) has an effect on the performance of the models. We train models on different combinations of visual and linguistic features and perform both (i) automatic evaluation of generated captions and (ii) examination of how useful different visual features are for the task of facial feature classification. Our results show that although original images encode the best possible representation for the task, the model trained on sketches can still perform relatively well. We also observe that augmenting the dataset with descriptions from a different domain can boost performance of the model. We conclude that face description generation systems are more susceptible to language rather than vision data augmentation. Overall, we demonstrate that face caption generation models display a strong imbalance in the utilisation of language and vision modalities, indicating a lack of proper information fusion. We also describe ethical implications of our study and argue that future work on human face description generation should create better, more representative datasets.
%U https://aclanthology.org/2022.pvlam-1.5
%P 26-40

Markdown (Informal)

[Examining the Effects of Language-and-Vision Data Augmentation for Generation of Descriptions of Human Faces](https://aclanthology.org/2022.pvlam-1.5) (Ilinykh et al., PVLAM 2022)

• Examining the Effects of Language-and-Vision Data Augmentation for Generation of Descriptions of Human Faces (Ilinykh et al., PVLAM 2022)

ACL

• Nikolai Ilinykh, Rafal Černiavski, Eva Elžbieta Sventickaitė, Viktorija Buzaitė, and Simon Dobnik. 2022. Examining the Effects of Language-and-Vision Data Augmentation for Generation of Descriptions of Human Faces. In Proceedings of the 2nd Workshop on People in Vision, Language, and the Mind, pages 26–40, Marseille, France. European Language Resources Association.