Qinlin Zhao
Author directory2026
Crowd Values: How Human Values Reshape LLM-Agent Communities
Xiangxu Zhang | Jiamin Wang | Qinlin Zhao | Hanze Guo | Linzhuo Li | Jing Yao | Xiaoyuan Yi | Xing Xie | Xiao Zhou
Findings of the Association for Computational Linguistics: EMNLP 2026
Xiangxu Zhang | Jiamin Wang | Qinlin Zhao | Hanze Guo | Linzhuo Li | Jing Yao | Xiaoyuan Yi | Xing Xie | Xiao Zhou
Findings of the Association for Computational Linguistics: EMNLP 2026
As LLMs evolve from standalone assistants into autonomous agents, they increasingly operate in populations where communication, coordination, and competition form community-like interaction networks. In such settings, individually value-steered behaviors can accumulate into group-level dynamics, raising the question of how human values shape collective outcomes in interacting LLM-agent populations. We introduce CIVA, a community-level value analysis framework that varies value direction and prevalence, then measures how these interventions propagate through repeated interactions into macro-level dynamics and micro-level behaviors. We apply it in a resource-constrained multi-agent community where LLM agents autonomously communicate, cooperate, and compete. The value scan reveals distinct structural response patterns: activation-supporting values such as benevolence supports persistence and resilience, an activation-constraining value such as tradition regulates growth and cohesion, and a bidirectionally disruptive value such as power perturbs collective regimes in both directions. At the micro level, representative interventions induce emergent behaviors including deception, betrayal, self-preservation, and power-seeking. These results show that human values can act as population-level structural variables in agent communities and motivate evaluation for multi-agent value alignment.
ScreenHaystack: Finding Blind Zones in GUI Grounding
Chenyue Li | Xiaoxiao Sun | Yubo Deng | Qinlin Zhao | Serena Yeung-Levy | Yuhui Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Chenyue Li | Xiaoxiao Sun | Yubo Deng | Qinlin Zhao | Serena Yeung-Levy | Yuhui Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
We introduce ScreenHaystack, a dynamic needle-in-a-haystack benchmark for evaluating spatial reliability in GUI grounding. Instead of testing each target at a fixed position, ScreenHaystack systematically relocates controlled target icons across high-resolution GUI backgrounds and measures whether models can localize them consistently. Using this benchmark, we find that leading GUI grounding models, including Qwen3-VL, UI-TARS, GTA, and UI-Venus, exhibit blind zones: spatial regions where grounding accuracy drops sharply despite fixed target appearance and instruction. These blind zones transfer to unseen ScreenSpot-Pro examples: targets inside blind zones are consistently harder to ground, with Qwen3-VL dropping by 16.1 percentage points, and controlled relocation shows that moving targets into blind zones decreases accuracy while moving them out improves accuracy. We further show through controlled synthetic experiments that uneven spatial coverage in training data can induce such blind zones. Therefore, we propose a simple strategy, blind-zone-oriented augmentation, which adds supervision in blind zones and improves ScreenSpot-Pro accuracy over both original and random augmented Click-100k fine-tuning.
Validity-Aware Assessment of LLM Value Orientation via Subset Item Response Theory
Qinlin Zhao | Xiaoyuan Yi | Jing Yao | Muhua Huang | Defu Lian | Xing Xie
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Qinlin Zhao | Xiaoyuan Yi | Jing Yao | Muhua Huang | Defu Lian | Xing Xie
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Measuring the value orientations of LLMs is helpful for diagnosing model risk and understanding their misalignment behavior. However, existing value benchmarks are often AI-generated and lack validity verification, leading to noisy and unreliable results. Inspired by Psychometrics, we propose a novel Subset Item Response Theory (S-IRT) model for assessing LLMs’ values under any value theories. S-IRT directly models the subset of values jointly expressed in each LLM response, rather than reducing responses to binary outcomes as in traditional IRT-based ability assessment. Meanwhile, it jointly estimates the LLM’s latent values and captures the relative priorities among them. Based on it, we further design a tailored information reward that estimates the information gain of each item in reducing uncertainty about the evaluated model’s value, and thus helps select a small set of high-quality items from existing benchmarks. Extensive experiments on 23 popular LLMs and three value systems (21 value dimensions) demonstrate S-IRT can be applied to multiple value systems simultaneously and improve evaluation reliability and validity. The information reward also helps reduce the number of questions to as few as 200 while preserving comparable evaluation effectiveness.
2025
Exploring the Choice Behavior of Large Language Models
Weidong Wu | Qinlin Zhao | Hao Chen | Lexin Zhou | Defu Lian | Hong Xie
Findings of the Association for Computational Linguistics: ACL 2025
Weidong Wu | Qinlin Zhao | Hao Chen | Lexin Zhou | Defu Lian | Hong Xie
Findings of the Association for Computational Linguistics: ACL 2025
Large Language Models (LLMs) are increasingly deployed as human assistants across various domains where they help to make choices. However, the mechanisms behind LLMs’ choice behavior remain unclear, posing risks in safety-critical situations. Inspired by the intrinsic and extrinsic motivation framework within the classic human behavioral model of Self-Determination Theory and its established research methodologies, we investigate the factors influencing LLMs’ choice behavior by constructing a virtual QA platform that includes three different experimental conditions, with four models from GPT and Llama series participating in repeated experiments. Our findings indicate that LLMs’ behavior is influenced not only by intrinsic attention bias but also by extrinsic social influence, exhibiting patterns similar to the Matthew effect and Conformity. We distinguish independent pathways of these two factors in LLMs’ behavior by self-report. This work provides new insights into understanding LLMs’ behavioral patterns, exploring their human-like characteristics.
2024
AgentReview: Exploring Peer Review Dynamics with LLM Agents
Yiqiao Jin | Qinlin Zhao | Yiyang Wang | Hao Chen | Kaijie Zhu | Yijia Xiao | Jindong Wang
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Yiqiao Jin | Qinlin Zhao | Yiyang Wang | Hao Chen | Kaijie Zhu | Yijia Xiao | Jindong Wang
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Peer review is fundamental to the integrity and advancement of scientific publication. Traditional methods of peer review analyses often rely on exploration and statistics of existing peer review data, which do not adequately address the multivariate nature of the process, account for the latent variables, and are further constrained by privacy concerns due to the sensitive nature of the data. We introduce AgentReview, the first large language model (LLM) based peer review simulation framework, which effectively disentangles the impacts of multiple latent factors and addresses the privacy issue. Our study reveals significant insights, including a notable 37.1% variation in paper decisions due to reviewers’ biases, supported by sociological theories such as the social influence theory, altruism fatigue, and authority bias. We believe that this study could offer valuable insights to improve the design of peer review mechanisms.