<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="feiyan-etal-2024-ji">
    <titleInfo>
        <title>基于中间层对齐的异构师生模型知识蒸馏(Knowledge distillation of heterogeneous teacher-student model with intermediate layer loss)</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Zhai</namePart>
        <namePart type="family">Feiyan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Wang</namePart>
        <namePart type="family">Renzhi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Li</namePart>
        <namePart type="family">Piji</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <language>
        <languageTerm type="text">zho</languageTerm>
    </language>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Maosong</namePart>
            <namePart type="family">Sun</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jiye</namePart>
            <namePart type="family">Liang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Xianpei</namePart>
            <namePart type="family">Han</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Zhiyuan</namePart>
            <namePart type="family">Liu</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yulan</namePart>
            <namePart type="family">He</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Chinese Information Processing Society of China</publisher>
            <place>
                <placeTerm type="text">Taiyuan, China</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>“知识蒸馏技术作为大语言模型时代的一项前沿模型压缩策略,通过将复杂模型的知识有效迁移至简单模型,显著降低了模型的参数规模和计算成本。尽管如此,目前主流的生成式大语言模型蒸馏算法主要集中于优化师生模型间的最后输出层损失,而忽视了对模型中间层的探索。此外,针对中间层蒸馏的研究往往对师生模型的结构一致性有着严格的要求,无法处理异构模型间的蒸馏问题,从而存在明显的局限性。针对这些问题,我们提出了一种新的知识蒸馏算法:引入了中间层蒸馏损失的异构生成式师生大语言模型知识蒸馏算法。该算法首先提取师生模型的中间层信息作为蒸馏对象,随后通过专门设计的中间层映射规则和对齐模块,实现异构模型间基于中间层的知识对齐与损失计算。最后,联合优化各个蒸馏损失的比例。通过在五个相关数据集上进行实验验证,我们的方法在提高蒸馏效果方面展现出显著优势。”</abstract>
    <identifier type="citekey">feiyan-etal-2024-ji</identifier>
    <location>
        <url>https://aclanthology.org/2024.ccl-1.71/</url>
    </location>
    <part>
        <date>2024-07</date>
        <extent unit="page">
            <start>910</start>
            <end>928</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T 基于中间层对齐的异构师生模型知识蒸馏(Knowledge distillation of heterogeneous teacher-student model with intermediate layer loss)
%A Feiyan, Zhai
%A Renzhi, Wang
%A Piji, Li
%Y Sun, Maosong
%Y Liang, Jiye
%Y Han, Xianpei
%Y Liu, Zhiyuan
%Y He, Yulan
%S Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference)
%D 2024
%8 July
%I Chinese Information Processing Society of China
%C Taiyuan, China
%G zho
%F feiyan-etal-2024-ji
%X “知识蒸馏技术作为大语言模型时代的一项前沿模型压缩策略,通过将复杂模型的知识有效迁移至简单模型,显著降低了模型的参数规模和计算成本。尽管如此,目前主流的生成式大语言模型蒸馏算法主要集中于优化师生模型间的最后输出层损失,而忽视了对模型中间层的探索。此外,针对中间层蒸馏的研究往往对师生模型的结构一致性有着严格的要求,无法处理异构模型间的蒸馏问题,从而存在明显的局限性。针对这些问题,我们提出了一种新的知识蒸馏算法:引入了中间层蒸馏损失的异构生成式师生大语言模型知识蒸馏算法。该算法首先提取师生模型的中间层信息作为蒸馏对象,随后通过专门设计的中间层映射规则和对齐模块,实现异构模型间基于中间层的知识对齐与损失计算。最后,联合优化各个蒸馏损失的比例。通过在五个相关数据集上进行实验验证,我们的方法在提高蒸馏效果方面展现出显著优势。”
%U https://aclanthology.org/2024.ccl-1.71/
%P 910-928

Markdown (Informal)

[基于中间层对齐的异构师生模型知识蒸馏(Knowledge distillation of heterogeneous teacher-student model with intermediate layer loss)](https://aclanthology.org/2024.ccl-1.71/) (Feiyan et al., CCL 2024)

• 基于中间层对齐的异构师生模型知识蒸馏(Knowledge distillation of heterogeneous teacher-student model with intermediate layer loss) (Feiyan et al., CCL 2024)

ACL

• Zhai Feiyan, Wang Renzhi, and Li Piji. 2024. 基于中间层对齐的异构师生模型知识蒸馏(Knowledge distillation of heterogeneous teacher-student model with intermediate layer loss). In Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference), pages 910–928, Taiyuan, China. Chinese Information Processing Society of China.