<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="lim-lauw-2023-large">
    <titleInfo>
        <title>Large-Scale Correlation Analysis of Automated Metrics for Topic Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Jia</namePart>
        <namePart type="given">Peng</namePart>
        <namePart type="family">Lim</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hady</namePart>
        <namePart type="family">Lauw</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Anna</namePart>
            <namePart type="family">Rogers</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jordan</namePart>
            <namePart type="family">Boyd-Graber</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Naoaki</namePart>
            <namePart type="family">Okazaki</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Toronto, Canada</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Automated coherence metrics constitute an important and popular way to evaluate topic models. Previous works present a mixed picture of their presumed correlation with human judgement. In this paper, we conduct a large-scale correlation analysis of coherence metrics. We propose a novel sampling approach to mine topics for the purpose of metric evaluation, and conduct the analysis via three large corpora showing that certain automated coherence metrics are correlated. Moreover, we extend the analysis to measure topical differences between corpora. Lastly, we examine the reliability of human judgement by conducting an extensive user study, which is designed as an amalgamation of different proxy tasks to derive a finer insight into the human decision-making processes. Our findings reveal some correlation between automated coherence metrics and human judgement, especially for generic corpora.</abstract>
    <identifier type="citekey">lim-lauw-2023-large</identifier>
    <identifier type="doi">10.18653/v1/2023.acl-long.776</identifier>
    <location>
        <url>https://aclanthology.org/2023.acl-long.776</url>
    </location>
    <part>
        <date>2023-07</date>
        <extent unit="page">
            <start>13874</start>
            <end>13898</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Large-Scale Correlation Analysis of Automated Metrics for Topic Models
%A Lim, Jia Peng
%A Lauw, Hady
%Y Rogers, Anna
%Y Boyd-Graber, Jordan
%Y Okazaki, Naoaki
%S Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
%D 2023
%8 July
%I Association for Computational Linguistics
%C Toronto, Canada
%F lim-lauw-2023-large
%X Automated coherence metrics constitute an important and popular way to evaluate topic models. Previous works present a mixed picture of their presumed correlation with human judgement. In this paper, we conduct a large-scale correlation analysis of coherence metrics. We propose a novel sampling approach to mine topics for the purpose of metric evaluation, and conduct the analysis via three large corpora showing that certain automated coherence metrics are correlated. Moreover, we extend the analysis to measure topical differences between corpora. Lastly, we examine the reliability of human judgement by conducting an extensive user study, which is designed as an amalgamation of different proxy tasks to derive a finer insight into the human decision-making processes. Our findings reveal some correlation between automated coherence metrics and human judgement, especially for generic corpora.
%R 10.18653/v1/2023.acl-long.776
%U https://aclanthology.org/2023.acl-long.776
%U https://doi.org/10.18653/v1/2023.acl-long.776
%P 13874-13898

Markdown (Informal)

[Large-Scale Correlation Analysis of Automated Metrics for Topic Models](https://aclanthology.org/2023.acl-long.776) (Lim & Lauw, ACL 2023)

• Large-Scale Correlation Analysis of Automated Metrics for Topic Models (Lim & Lauw, ACL 2023)

ACL

• Jia Peng Lim and Hady Lauw. 2023. Large-Scale Correlation Analysis of Automated Metrics for Topic Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13874–13898, Toronto, Canada. Association for Computational Linguistics.