<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="fekete-bjerva-2025-linguistically">
    <titleInfo>
        <title>Linguistically Grounded Analysis of Language Models using Shapley Head Values</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Marcell</namePart>
        <namePart type="family">Fekete</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Johannes</namePart>
        <namePart type="family">Bjerva</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-04</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: NAACL 2025</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Luis</namePart>
            <namePart type="family">Chiruzzo</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alan</namePart>
            <namePart type="family">Ritter</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lu</namePart>
            <namePart type="family">Wang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Albuquerque, New Mexico</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-195-7</identifier>
    </relatedItem>
    <abstract>Understanding how linguistic knowledge is encoded in language models is crucial for improving their generalisation capabilities. In this paper, we investigate the processing of morphosyntactic phenomena, by leveraging a recently proposed method for probing language models via Shapley Head Values (SHVs). Using the English language BLiMP dataset, we test our approach on two widely used models, BERT and RoBERTa, and compare how linguistic constructions such as anaphor agreement and filler-gap dependencies are handled. Through quantitative pruning and qualitative clustering analysis, we demonstrate that attention heads responsible for processing related linguistic phenomena cluster together. Our results show that SHV-based attributions reveal distinct patterns across both models, providing insights into how language models organize and process linguistic information. These findings support the hypothesis that language models learn subnetworks corresponding to linguistic theory, with potential implications for cross-linguistic model analysis and interpretability in Natural Language Processing (NLP).</abstract>
    <identifier type="citekey">fekete-bjerva-2025-linguistically</identifier>
    <identifier type="doi">10.18653/v1/2025.findings-naacl.49</identifier>
    <location>
        <url>https://aclanthology.org/2025.findings-naacl.49/</url>
    </location>
    <part>
        <date>2025-04</date>
        <extent unit="page">
            <start>850</start>
            <end>865</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Linguistically Grounded Analysis of Language Models using Shapley Head Values
%A Fekete, Marcell
%A Bjerva, Johannes
%Y Chiruzzo, Luis
%Y Ritter, Alan
%Y Wang, Lu
%S Findings of the Association for Computational Linguistics: NAACL 2025
%D 2025
%8 April
%I Association for Computational Linguistics
%C Albuquerque, New Mexico
%@ 979-8-89176-195-7
%F fekete-bjerva-2025-linguistically
%X Understanding how linguistic knowledge is encoded in language models is crucial for improving their generalisation capabilities. In this paper, we investigate the processing of morphosyntactic phenomena, by leveraging a recently proposed method for probing language models via Shapley Head Values (SHVs). Using the English language BLiMP dataset, we test our approach on two widely used models, BERT and RoBERTa, and compare how linguistic constructions such as anaphor agreement and filler-gap dependencies are handled. Through quantitative pruning and qualitative clustering analysis, we demonstrate that attention heads responsible for processing related linguistic phenomena cluster together. Our results show that SHV-based attributions reveal distinct patterns across both models, providing insights into how language models organize and process linguistic information. These findings support the hypothesis that language models learn subnetworks corresponding to linguistic theory, with potential implications for cross-linguistic model analysis and interpretability in Natural Language Processing (NLP).
%R 10.18653/v1/2025.findings-naacl.49
%U https://aclanthology.org/2025.findings-naacl.49/
%U https://doi.org/10.18653/v1/2025.findings-naacl.49
%P 850-865

Markdown (Informal)

[Linguistically Grounded Analysis of Language Models using Shapley Head Values](https://aclanthology.org/2025.findings-naacl.49/) (Fekete & Bjerva, Findings 2025)

• Linguistically Grounded Analysis of Language Models using Shapley Head Values (Fekete & Bjerva, Findings 2025)

ACL

• Marcell Fekete and Johannes Bjerva. 2025. Linguistically Grounded Analysis of Language Models using Shapley Head Values. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 850–865, Albuquerque, New Mexico. Association for Computational Linguistics.