<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="laverghetta-jr-etal-2021-transformer">
    <titleInfo>
        <title>Can Transformer Language Models Predict Psychometric Properties?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Antonio</namePart>
        <namePart type="family">Laverghetta Jr.</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Animesh</namePart>
        <namePart type="family">Nighojkar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jamshidbek</namePart>
        <namePart type="family">Mirzakhalov</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">John</namePart>
        <namePart type="family">Licato</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Lun-Wei</namePart>
            <namePart type="family">Ku</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Vivi</namePart>
            <namePart type="family">Nastase</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ivan</namePart>
            <namePart type="family">Vulić</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Transformer-based language models (LMs) continue to advance state-of-the-art performance on NLP benchmark tasks, including tasks designed to mimic human-inspired “commonsense” competencies. To better understand the degree to which LMs can be said to have certain linguistic reasoning skills, researchers are beginning to adapt the tools and concepts of the field of psychometrics. But to what extent can the benefits flow in the other direction? I.e., can LMs be of use in predicting what the psychometric properties of test items will be when those items are given to human participants? We gather responses from numerous human participants and LMs (transformer- and non-transformer-based) on a broad diagnostic test of linguistic competencies. We then use the responses to calculate standard psychometric properties of the items in the diagnostic test, using the human responses and the LM responses separately. We then determine how well these two sets of predictions match. We find cases in which transformer-based LMs predict psychometric properties consistently well in certain categories but consistently poorly in others, thus providing new insights into fundamental similarities and differences between human and LM reasoning.</abstract>
    <identifier type="citekey">laverghetta-jr-etal-2021-transformer</identifier>
    <identifier type="doi">10.18653/v1/2021.starsem-1.2</identifier>
    <location>
        <url>https://aclanthology.org/2021.starsem-1.2</url>
    </location>
    <part>
        <date>2021-08</date>
        <extent unit="page">
            <start>12</start>
            <end>25</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Can Transformer Language Models Predict Psychometric Properties?
%A Laverghetta Jr., Antonio
%A Nighojkar, Animesh
%A Mirzakhalov, Jamshidbek
%A Licato, John
%Y Ku, Lun-Wei
%Y Nastase, Vivi
%Y Vulić, Ivan
%S Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics
%D 2021
%8 August
%I Association for Computational Linguistics
%C Online
%F laverghetta-jr-etal-2021-transformer
%X Transformer-based language models (LMs) continue to advance state-of-the-art performance on NLP benchmark tasks, including tasks designed to mimic human-inspired “commonsense” competencies. To better understand the degree to which LMs can be said to have certain linguistic reasoning skills, researchers are beginning to adapt the tools and concepts of the field of psychometrics. But to what extent can the benefits flow in the other direction? I.e., can LMs be of use in predicting what the psychometric properties of test items will be when those items are given to human participants? We gather responses from numerous human participants and LMs (transformer- and non-transformer-based) on a broad diagnostic test of linguistic competencies. We then use the responses to calculate standard psychometric properties of the items in the diagnostic test, using the human responses and the LM responses separately. We then determine how well these two sets of predictions match. We find cases in which transformer-based LMs predict psychometric properties consistently well in certain categories but consistently poorly in others, thus providing new insights into fundamental similarities and differences between human and LM reasoning.
%R 10.18653/v1/2021.starsem-1.2
%U https://aclanthology.org/2021.starsem-1.2
%U https://doi.org/10.18653/v1/2021.starsem-1.2
%P 12-25

Markdown (Informal)

[Can Transformer Language Models Predict Psychometric Properties?](https://aclanthology.org/2021.starsem-1.2) (Laverghetta Jr. et al., *SEM 2021)

• Can Transformer Language Models Predict Psychometric Properties? (Laverghetta Jr. et al., *SEM 2021)

ACL

• Antonio Laverghetta Jr., Animesh Nighojkar, Jamshidbek Mirzakhalov, and John Licato. 2021. Can Transformer Language Models Predict Psychometric Properties?. In Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics, pages 12–25, Online. Association for Computational Linguistics.