<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="yang-etal-2025-llm">
    <titleInfo>
        <title>LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Xiaohao</namePart>
        <namePart type="family">Yang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">He</namePart>
        <namePart type="family">Zhao</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Dinh</namePart>
        <namePart type="family">Phung</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Wray</namePart>
        <namePart type="family">Buntine</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Lan</namePart>
        <namePart type="family">Du</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Topic modeling has been a widely used tool for unsupervised text analysis. However, comprehensive evaluations of a topic model remain challenging. Existing evaluation methods are either less comparable across different models (e.g., perplexity) or focus on only one specific aspect of a model (e.g., topic quality or document representation quality) at a time, which is insufficient to reflect the overall model performance. In this paper, we propose WALM (Word Agreement with Language Model), a new evaluation method for topic modeling that considers the semantic quality of document representations and topics in a joint manner, leveraging the power of Large Language Models (LLMs). With extensive experiments involving different types of topic models, WALM is shown to align with human judgment and can serve as a complementary evaluation method to the existing ones, bringing a new perspective to topic modeling. Our software package is available at https://github.com/Xiaohao-Yang/Topic_Model_Evaluation.</abstract>
    <identifier type="citekey">yang-etal-2025-llm</identifier>
    <identifier type="doi">10.1162/tacl_a_00744</identifier>
    <location>
        <url>https://aclanthology.org/2025.tacl-1.17/</url>
    </location>
    <part>
        <date>2025</date>
        <detail type="volume"><number>13</number></detail>
        <extent unit="page">
            <start>357</start>
            <end>375</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models
%A Yang, Xiaohao
%A Zhao, He
%A Phung, Dinh
%A Buntine, Wray
%A Du, Lan
%J Transactions of the Association for Computational Linguistics
%D 2025
%V 13
%I MIT Press
%C Cambridge, MA
%F yang-etal-2025-llm
%X Topic modeling has been a widely used tool for unsupervised text analysis. However, comprehensive evaluations of a topic model remain challenging. Existing evaluation methods are either less comparable across different models (e.g., perplexity) or focus on only one specific aspect of a model (e.g., topic quality or document representation quality) at a time, which is insufficient to reflect the overall model performance. In this paper, we propose WALM (Word Agreement with Language Model), a new evaluation method for topic modeling that considers the semantic quality of document representations and topics in a joint manner, leveraging the power of Large Language Models (LLMs). With extensive experiments involving different types of topic models, WALM is shown to align with human judgment and can serve as a complementary evaluation method to the existing ones, bringing a new perspective to topic modeling. Our software package is available at https://github.com/Xiaohao-Yang/Topic_Model_Evaluation.
%R 10.1162/tacl_a_00744
%U https://aclanthology.org/2025.tacl-1.17/
%U https://doi.org/10.1162/tacl_a_00744
%P 357-375

Markdown (Informal)

[LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models](https://aclanthology.org/2025.tacl-1.17/) (Yang et al., TACL 2025)

• LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models (Yang et al., TACL 2025)

ACL

• Xiaohao Yang, He Zhao, Dinh Phung, Wray Buntine, and Lan Du. 2025. LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models. Transactions of the Association for Computational Linguistics, 13:357–375.