<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="lee-etal-2026-knowsim">
    <titleInfo>
        <title>KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Yoonjoo</namePart>
        <namePart type="family">Lee</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hyoungwook</namePart>
        <namePart type="family">Jin</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Tae</namePart>
        <namePart type="given">Soo</namePart>
        <namePart type="family">Kim</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Shaoyang</namePart>
        <namePart type="family">Zhang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Philippe</namePart>
        <namePart type="family">Laban</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Q.Vera</namePart>
        <namePart type="family">Liao</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-459-0</identifier>
    </relatedItem>
    <abstract>To effectively collaborate with users on knowledge-intensive tasks, Large Language Models (LLMs) must perform information calibration: matching content to a user’s evolving understanding and cognitive capacity. Yet user simulators used to evaluate and train LLMs do not explicitly model user knowledge, so they neither produce realistic interactions across knowledge levels nor reflect how interactions unfold as that knowledge evolves. To close this gap, we introduce KnowSim, an evaluation framework built around a user simulator that maintains explicit knowledge states—a graph of Information Units with prerequisite relationships—that evolves under update rules grounded in learning theory. KnowSim computes three metrics (Knowledge Gain, Delivery Calibration, Cognitive Overload) directly from the knowledge state trajectory, reflecting key mechanistic aspects of information calibration. We validate KnowSim against 705 human–AI sessions across two domains, stratified by knowledge level: its rankings align significantly with human judgments (73–74% sign agreement), outperforming three baseline simulators. Applied to 8 frontier LLMs, KnowSim reveals that the best model shifts by user knowledge level—aptitude–treatment interactions invisible to standard evaluation.</abstract>
    <identifier type="citekey">lee-etal-2026-knowsim</identifier>
    <location>
        <url>https://aclanthology.org/2026.emnlp-main.1307/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>29750</start>
            <end>29780</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn
%A Lee, Yoonjoo
%A Jin, Hyoungwook
%A Kim, Tae Soo
%A Zhang, Shaoyang
%A Laban, Philippe
%A Liao, Q.Vera
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F lee-etal-2026-knowsim
%X To effectively collaborate with users on knowledge-intensive tasks, Large Language Models (LLMs) must perform information calibration: matching content to a user’s evolving understanding and cognitive capacity. Yet user simulators used to evaluate and train LLMs do not explicitly model user knowledge, so they neither produce realistic interactions across knowledge levels nor reflect how interactions unfold as that knowledge evolves. To close this gap, we introduce KnowSim, an evaluation framework built around a user simulator that maintains explicit knowledge states—a graph of Information Units with prerequisite relationships—that evolves under update rules grounded in learning theory. KnowSim computes three metrics (Knowledge Gain, Delivery Calibration, Cognitive Overload) directly from the knowledge state trajectory, reflecting key mechanistic aspects of information calibration. We validate KnowSim against 705 human–AI sessions across two domains, stratified by knowledge level: its rankings align significantly with human judgments (73–74% sign agreement), outperforming three baseline simulators. Applied to 8 frontier LLMs, KnowSim reveals that the best model shifts by user knowledge level—aptitude–treatment interactions invisible to standard evaluation.
%U https://aclanthology.org/2026.emnlp-main.1307/
%P 29750-29780

Markdown (Informal)

[KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn](https://aclanthology.org/2026.emnlp-main.1307/) (Lee et al., EMNLP 2026)

• KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn (Lee et al., EMNLP 2026)

ACL

• Yoonjoo Lee, Hyoungwook Jin, Tae Soo Kim, Shaoyang Zhang, Philippe Laban, and Q.Vera Liao. 2026. KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learn. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 29750–29780, Budapest, Hungary. Association for Computational Linguistics.