Danping Yan
Author directory2026
Beyond Warmth: Risk-Tiered Auditing of Credibility in LLM Companion Responses
Danping Yan
Proceedings of the 3rd Workshop on Language Understanding in the Human-Machine Era (LUHME 2026)
Danping Yan
Proceedings of the 3rd Workshop on Language Understanding in the Human-Machine Era (LUHME 2026)
Large language models (LLMs) are increasingly used as conversational companions, where warm and supportive responses may still become poorly scoped as guidance under uncertainty or risk. Existing evaluations often emphasize affective alignment, helpfulness, persona consistency, or overt safety violations, leaving less attention to failures where caring language becomes action-relevant but insufficiently bounded. We introduce a risk-tiered audit framework for credibility under risk in supportive companion responses, evaluating whether responses remain responsible, bounded, and behaviorally appropriate when interpreted as guidance. The framework combines fixed scenario-level risk tiers, five human-facing diagnostic credibility dimensions, and response-level failure tags to identify when warm support becomes poorly scoped guidance. Risk tiers indicate where mis-scoped guidance would be more consequential, while credibility scores and failure tags characterize response behavior in those contexts. Across a heterogeneous response pool, we find a recurring tiered failure pattern: HIGH-tier scenarios selectively expose responsibility- and behavior-facing failures even when tone and identity remain comparatively stable. These failures are dominated by overconfident guidance, where models discourage qualified escalation or commit too confidently to action under uncertainty. Cross-judge diagnostics, human validation, and portability analyses support a conservative interpretation of this credibility signal. Overall, the results suggest that supportive dialogue systems should be evaluated not only for warmth, but also for whether support remains responsible, bounded, and well-scoped when guidance carries higher stakes.
GROWTHEVAL: Evaluating Stage-Conditioned Developmental Realism in Role-Playing LLMs
Danping Yan
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Danping Yan
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Large language models (LLMs) are increasingly used as role-playing dialogue agents in long-horizon applications such as tutoring, companionship, and narrative interaction, where users may expect stable persona consistency as well as plausible stage-conditioned change. Existing evaluations emphasize static persona fidelity, temporal factuality, or isolated social skills, but rarely test whether role-playing models produce coherent developmental differentiation across ordered stages. We introduce GROWTHEVAL, a psychology-inspired diagnostic framework for assessing stage-conditioned developmental realism in single-turn, stage-isolated role-playing responses. Using stage-conditioned scenarios and multi-axis scoring, GrowthEval evaluates cross-stage patterns in developmental appropriateness, role-play coherence, and canon fidelity across five socio-emotional dimensions. Across the evaluated LLMs, we observe a model-tiered empirical pattern: higher-capacity models respond more consistently to stage conditioning but can over-mature at early stages, whereas lower-capacity models exhibit flatter and weakly differentiated profiles. Reliability and robustness analyses provide evidence for the stability of these findings. Overall, the evaluated LLMs only partially realize coherent, stage-appropriate developmental change. GROWTHEVAL provides an extensible diagnostic framework for analyzing stage-conditioned behavioral differentiation and associated failure modes in single-turn role-playing responses.