<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="phukan-etal-2024-peering">
    <titleInfo>
        <title>Peering into the Mind of Language Models: An Approach for Attribution in Contextual Question Answering</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Anirudh</namePart>
        <namePart type="family">Phukan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Shwetha</namePart>
        <namePart type="family">Somasundaram</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Apoorv</namePart>
        <namePart type="family">Saxena</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Koustava</namePart>
        <namePart type="family">Goswami</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Balaji</namePart>
        <namePart type="given">Vasan</namePart>
        <namePart type="family">Srinivasan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Lun-Wei</namePart>
            <namePart type="family">Ku</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Andre</namePart>
            <namePart type="family">Martins</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Vivek</namePart>
            <namePart type="family">Srikumar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Bangkok, Thailand</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>With the enhancement in the field of generative artificial intelligence (AI), contextual question answering has become extremely relevant. Attributing model generations to the input source document is essential to ensure trustworthiness and reliability. We observe that when large language models (LLMs) are used for contextual question answering, the output answer often consists of text copied verbatim from the input prompt which is linked together with “glue text” generated by the LLM. Motivated by this, we propose that LLMs have an inherent awareness from where the text was copied, likely captured in the hidden states of the LLM. We introduce a novel method for attribution in contextual question answering, leveraging the hidden state representations of LLMs. Our approach bypasses the need for extensive model retraining and retrieval model overhead, offering granular attributions and preserving the quality of generated answers. Our experimental results demonstrate that our method performs on par or better than GPT-4 at identifying verbatim copied segments in LLM generations and in attributing these segments to their source. Importantly, our method shows robust performance across various LLM architectures, highlighting its broad applicability. Additionally, we present Verifiability-granular, an attribution dataset which has token level annotations for LLM generations in the contextual question answering setup.</abstract>
    <identifier type="citekey">phukan-etal-2024-peering</identifier>
    <identifier type="doi">10.18653/v1/2024.findings-acl.682</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-acl.682/</url>
    </location>
    <part>
        <date>2024-08</date>
        <extent unit="page">
            <start>11481</start>
            <end>11495</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Peering into the Mind of Language Models: An Approach for Attribution in Contextual Question Answering
%A Phukan, Anirudh
%A Somasundaram, Shwetha
%A Saxena, Apoorv
%A Goswami, Koustava
%A Srinivasan, Balaji Vasan
%Y Ku, Lun-Wei
%Y Martins, Andre
%Y Srikumar, Vivek
%S Findings of the Association for Computational Linguistics: ACL 2024
%D 2024
%8 August
%I Association for Computational Linguistics
%C Bangkok, Thailand
%F phukan-etal-2024-peering
%X With the enhancement in the field of generative artificial intelligence (AI), contextual question answering has become extremely relevant. Attributing model generations to the input source document is essential to ensure trustworthiness and reliability. We observe that when large language models (LLMs) are used for contextual question answering, the output answer often consists of text copied verbatim from the input prompt which is linked together with “glue text” generated by the LLM. Motivated by this, we propose that LLMs have an inherent awareness from where the text was copied, likely captured in the hidden states of the LLM. We introduce a novel method for attribution in contextual question answering, leveraging the hidden state representations of LLMs. Our approach bypasses the need for extensive model retraining and retrieval model overhead, offering granular attributions and preserving the quality of generated answers. Our experimental results demonstrate that our method performs on par or better than GPT-4 at identifying verbatim copied segments in LLM generations and in attributing these segments to their source. Importantly, our method shows robust performance across various LLM architectures, highlighting its broad applicability. Additionally, we present Verifiability-granular, an attribution dataset which has token level annotations for LLM generations in the contextual question answering setup.
%R 10.18653/v1/2024.findings-acl.682
%U https://aclanthology.org/2024.findings-acl.682/
%U https://doi.org/10.18653/v1/2024.findings-acl.682
%P 11481-11495

Markdown (Informal)

[Peering into the Mind of Language Models: An Approach for Attribution in Contextual Question Answering](https://aclanthology.org/2024.findings-acl.682/) (Phukan et al., Findings 2024)

• Peering into the Mind of Language Models: An Approach for Attribution in Contextual Question Answering (Phukan et al., Findings 2024)

ACL

• Anirudh Phukan, Shwetha Somasundaram, Apoorv Saxena, Koustava Goswami, and Balaji Vasan Srinivasan. 2024. Peering into the Mind of Language Models: An Approach for Attribution in Contextual Question Answering. In Findings of the Association for Computational Linguistics: ACL 2024, pages 11481–11495, Bangkok, Thailand. Association for Computational Linguistics.