<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="malkiel-wolf-2021-maximal">
    <titleInfo>
        <title>Maximal Multiverse Learning for Promoting Cross-Task Generalization of Fine-Tuned Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Itzik</namePart>
        <namePart type="family">Malkiel</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Lior</namePart>
        <namePart type="family">Wolf</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-04</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Paola</namePart>
            <namePart type="family">Merlo</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jorg</namePart>
            <namePart type="family">Tiedemann</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Reut</namePart>
            <namePart type="family">Tsarfaty</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Language modeling with BERT consists of two phases of (i) unsupervised pre-training on unlabeled text, and (ii) fine-tuning for a specific supervised task. We present a method that leverages the second phase to its fullest, by applying an extensive number of parallel classifier heads, which are enforced to be orthogonal, while adaptively eliminating the weaker heads during training. We conduct an extensive inter- and intra-dataset evaluation, showing that our method improves the generalization ability of BERT, sometimes leading to a +9% gain in accuracy. These results highlight the importance of a proper fine-tuning procedure, especially for relatively smaller-sized datasets. Our code is attached as supplementary.</abstract>
    <identifier type="citekey">malkiel-wolf-2021-maximal</identifier>
    <identifier type="doi">10.18653/v1/2021.eacl-main.14</identifier>
    <location>
        <url>https://aclanthology.org/2021.eacl-main.14</url>
    </location>
    <part>
        <date>2021-04</date>
        <extent unit="page">
            <start>187</start>
            <end>199</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Maximal Multiverse Learning for Promoting Cross-Task Generalization of Fine-Tuned Language Models
%A Malkiel, Itzik
%A Wolf, Lior
%Y Merlo, Paola
%Y Tiedemann, Jorg
%Y Tsarfaty, Reut
%S Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
%D 2021
%8 April
%I Association for Computational Linguistics
%C Online
%F malkiel-wolf-2021-maximal
%X Language modeling with BERT consists of two phases of (i) unsupervised pre-training on unlabeled text, and (ii) fine-tuning for a specific supervised task. We present a method that leverages the second phase to its fullest, by applying an extensive number of parallel classifier heads, which are enforced to be orthogonal, while adaptively eliminating the weaker heads during training. We conduct an extensive inter- and intra-dataset evaluation, showing that our method improves the generalization ability of BERT, sometimes leading to a +9% gain in accuracy. These results highlight the importance of a proper fine-tuning procedure, especially for relatively smaller-sized datasets. Our code is attached as supplementary.
%R 10.18653/v1/2021.eacl-main.14
%U https://aclanthology.org/2021.eacl-main.14
%U https://doi.org/10.18653/v1/2021.eacl-main.14
%P 187-199

Markdown (Informal)

[Maximal Multiverse Learning for Promoting Cross-Task Generalization of Fine-Tuned Language Models](https://aclanthology.org/2021.eacl-main.14) (Malkiel & Wolf, EACL 2021)

• Maximal Multiverse Learning for Promoting Cross-Task Generalization of Fine-Tuned Language Models (Malkiel & Wolf, EACL 2021)

ACL

• Itzik Malkiel and Lior Wolf. 2021. Maximal Multiverse Learning for Promoting Cross-Task Generalization of Fine-Tuned Language Models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 187–199, Online. Association for Computational Linguistics.