Hanjia Lyu
Author directory2026
Seeing the Poem: Image-Semantic Detection of AI-Generated Modern Chinese Poetry with MLLMs
Shanshan Wang | Fengying Ye | Hanjia Lyu | Caiwen Gou | Junchao Wu | Jingming Yao | Cheng-zhong Xu | Jiebo Luo | Derek F. Wong
Findings of the Association for Computational Linguistics: EMNLP 2026
Shanshan Wang | Fengying Ye | Hanjia Lyu | Caiwen Gou | Junchao Wu | Jingming Yao | Cheng-zhong Xu | Jiebo Luo | Derek F. Wong
Findings of the Association for Computational Linguistics: EMNLP 2026
Previous detection studies have shown that LLMs cannot be effectively used as detectors, but these studies have not addressed modern Chinese poetry. Moreover, no relevant research has explored the performance of LLMs in detecting modern Chinese poetry. This paper evaluates and enhances the performance of LLMs as detectors for modern Chinese poetry, and proposes an image-semantic guided poetry detection method. Compared with traditional detection approaches, our method innovatively incorporates images that reflect the content of the poetry. Through example-driven approaches, our method effectively integrates information such as meaning, imagery, and feeling from the image, then forms a complementary judgment with the poem text. Experimental results demonstrate that the LLM detectors based on our method outperform baseline detectors based on plain text, and even surpass the best-performing traditional detector, RoBERTa. The Gemini detector using our method achieves a Macro-F1 score of 85.65%, reaching the state-of-the-art level. The performance improvements of different LLM detectors on multiple LLMs-generated data prove the effectiveness of our method.
Revising Context, Shifting Simulated Stance: Auditing LLM-Based Stance Simulation in Online Discussions
Xinnong Zhang | Wanting Shan | Hanjia Lyu | Zhongyu Wei | Jiebo Luo
Findings of the Association for Computational Linguistics: EMNLP 2026
Xinnong Zhang | Wanting Shan | Hanjia Lyu | Zhongyu Wei | Jiebo Luo
Findings of the Association for Computational Linguistics: EMNLP 2026
Large language models are increasingly used to simulate social media users and infer how individuals may respond to online discussions. However, it remains unclear whether these simulations reflect precise user-specific beliefs or whether they are highly sensitive to semantically independent changes in conversational contexts. In this work, we study counterfactual context revision as a framework for auditing LLM-based stance simulation. Given an original online conversation, we first infer a target user’s stance toward a specific topic. We then apply controlled revision strategies to the conversational context and simulate the user’s stance again under the revised context. We compare text-only revision strategies with a multimodal one that incorporates meme-based context and evaluate two main effectiveness metrics, i.e., average directional stance shift and stance transition rate. The results reveal effective and robust stance transitions in both text-only and multimodal strategies across different polarization-preference mechanisms. Our study contributes an evaluation framework for understanding the context sensitivity of LLM-based stance simulation (https://github.com/STARResearchLab/StanceShift). More broadly, it highlights both the promise and risk of using LLMs as proxies for social media users in social simulation.
2025
GUI Agents: A Survey
Dang Nguyen | Jian Chen | Yu Wang | Gang Wu | Namyong Park | Zhengmian Hu | Hanjia Lyu | Junda Wu | Ryan Aponte | Yu Xia | Xintong Li | Jing Shi | Hongjie Chen | Viet Dac Lai | Zhouhang Xie | Sungchul Kim | Ruiyi Zhang | Tong Yu | Mehrab Tanjim | Nesreen K. Ahmed | Puneet Mathur | Seunghyun Yoon | Lina Yao | Branislav Kveton | Jihyung Kil | Thien Huu Nguyen | Trung Bui | Tianyi Zhou | Ryan A. Rossi | Franck Dernoncourt
Findings of the Association for Computational Linguistics: ACL 2025
Dang Nguyen | Jian Chen | Yu Wang | Gang Wu | Namyong Park | Zhengmian Hu | Hanjia Lyu | Junda Wu | Ryan Aponte | Yu Xia | Xintong Li | Jing Shi | Hongjie Chen | Viet Dac Lai | Zhouhang Xie | Sungchul Kim | Ruiyi Zhang | Tong Yu | Mehrab Tanjim | Nesreen K. Ahmed | Puneet Mathur | Seunghyun Yoon | Lina Yao | Branislav Kveton | Jihyung Kil | Thien Huu Nguyen | Trung Bui | Tianyi Zhou | Ryan A. Rossi | Franck Dernoncourt
Findings of the Association for Computational Linguistics: ACL 2025
Graphical User Interface (GUI) agents, powered by Large Foundation Models, have emerged as a transformative approach to automating human-computer interaction. These agents autonomously interact with digital systems via GUIs, emulating human actions such as clicking, typing, and navigating visual elements across diverse platforms. Motivated by the growing interest and fundamental importance of GUI agents, we provide a comprehensive survey that categorizes their benchmarks, evaluation metrics, architectures, and training methods. We propose a unified framework that delineates their perception, reasoning, planning, and acting capabilities. Furthermore, we identify important open challenges and discuss key future directions. Finally, this work serves as a basis for practitioners and researchers to gain an intuitive understanding of current progress, techniques, benchmarks, and critical open problems that remain to be addressed.
Evolver: Chain-of-Evolution Prompting to Boost Large Multimodal Models for Hateful Meme Detection
Jinfa Huang | Jinsheng Pan | Zhongwei Wan | Hanjia Lyu | Jiebo Luo
Proceedings of the 31st International Conference on Computational Linguistics
Jinfa Huang | Jinsheng Pan | Zhongwei Wan | Hanjia Lyu | Jiebo Luo
Proceedings of the 31st International Conference on Computational Linguistics
Hateful memes continuously evolve as new ones emerge by blending progressive cultural ideas, rendering existing methods that rely on extensive training obsolete or ineffective. In this work, we propose Evolver, which incorporates Large Multimodal Models (LMMs) via Chain-of-Evolution (CoE) Prompting, by integrating the evolution attribute and in-context information of memes. Specifically, Evolver simulates the evolving and expressing process of memes and reasons through LMMs in a step-by-step manner using an evolutionary pair mining module, an evolutionary information extractor, and a contextual relevance amplifier. Extensive experiments on public FHM, MAMI, and HarM datasets show that CoE prompting can be incorporated into existing LMMs to improve their performance. More encouragingly, it can serve as an interpretive tool to promote the understanding of the evolution of memes.
From Selection to Generation: A Survey of LLM-based Active Learning
Yu Xia | Subhojyoti Mukherjee | Zhouhang Xie | Junda Wu | Xintong Li | Ryan Aponte | Hanjia Lyu | Joe Barrow | Hongjie Chen | Franck Dernoncourt | Branislav Kveton | Tong Yu | Ruiyi Zhang | Jiuxiang Gu | Nesreen K. Ahmed | Yu Wang | Xiang Chen | Hanieh Deilamsalehy | Sungchul Kim | Zhengmian Hu | Yue Zhao | Nedim Lipka | Seunghyun Yoon | Ting-Hao Kenneth Huang | Zichao Wang | Puneet Mathur | Soumyabrata Pal | Koyel Mukherjee | Zhehao Zhang | Namyong Park | Thien Huu Nguyen | Jiebo Luo | Ryan A. Rossi | Julian McAuley
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Yu Xia | Subhojyoti Mukherjee | Zhouhang Xie | Junda Wu | Xintong Li | Ryan Aponte | Hanjia Lyu | Joe Barrow | Hongjie Chen | Franck Dernoncourt | Branislav Kveton | Tong Yu | Ruiyi Zhang | Jiuxiang Gu | Nesreen K. Ahmed | Yu Wang | Xiang Chen | Hanieh Deilamsalehy | Sungchul Kim | Zhengmian Hu | Yue Zhao | Nedim Lipka | Seunghyun Yoon | Ting-Hao Kenneth Huang | Zichao Wang | Puneet Mathur | Soumyabrata Pal | Koyel Mukherjee | Zhehao Zhang | Namyong Park | Thien Huu Nguyen | Jiebo Luo | Ryan A. Rossi | Julian McAuley
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Active Learning (AL) has been a powerful paradigm for improving model efficiency and performance by selecting the most informative data points for labeling and training. In recent active learning frameworks, Large Language Models (LLMs) have been employed not only for selection but also for generating entirely new data instances and providing more cost-effective annotations. Motivated by the increasing importance of high-quality data and efficient model training in the era of LLMs, we present a comprehensive survey on LLM-based Active Learning. We introduce an intuitive taxonomy that categorizes these techniques and discuss the transformative roles LLMs can play in the active learning loop. We further examine the impact of AL on LLM learning paradigms and its applications across various domains. Finally, we identify open challenges and propose future research directions. This survey aims to serve as an up-to-date resource for researchers and practitioners seeking to gain an intuitive understanding of LLM-based AL techniques and deploy them to new applications.
2024
LLM-Rec: Personalized Recommendation via Prompting Large Language Models
Hanjia Lyu | Song Jiang | Hanqing Zeng | Yinglong Xia | Qifan Wang | Si Zhang | Ren Chen | Chris Leung | Jiajie Tang | Jiebo Luo
Findings of the Association for Computational Linguistics: NAACL 2024
Hanjia Lyu | Song Jiang | Hanqing Zeng | Yinglong Xia | Qifan Wang | Si Zhang | Ren Chen | Chris Leung | Jiajie Tang | Jiebo Luo
Findings of the Association for Computational Linguistics: NAACL 2024
Text-based recommendation holds a wide range of practical applications due to its versatility, as textual descriptions can represent nearly any type of item. However, directly employing the original item descriptions may not yield optimal recommendation performance due to the lack of comprehensive information to align with user preferences. Recent advances in large language models (LLMs) have showcased their remarkable ability to harness commonsense knowledge and reasoning. In this study, we introduce a novel approach, coined LLM-Rec, which incorporates four distinct prompting strategies of text enrichment for improving personalized text-based recommendations. Our empirical experiments reveal that using LLM-augmented text significantly enhances recommendation quality. Even basic MLP (Multi-Layer Perceptron) models achieve comparable or even better results than complex content-based methods. Notably, the success of LLM-Rec lies in its prompting strategies, which effectively tap into the language model’s comprehension of both general and specific item characteristics. This highlights the importance of employing diverse prompts and input augmentation techniques to boost the recommendation effectiveness of LLMs.
SoMeLVLM: A Large Vision Language Model for Social Media Processing
Xinnong Zhang | Haoyu Kuang | Xinyi Mou | Hanjia Lyu | Kun Wu | Siming Chen | Jiebo Luo | Xuanjing Huang | Zhongyu Wei
Findings of the Association for Computational Linguistics: ACL 2024
Xinnong Zhang | Haoyu Kuang | Xinyi Mou | Hanjia Lyu | Kun Wu | Siming Chen | Jiebo Luo | Xuanjing Huang | Zhongyu Wei
Findings of the Association for Computational Linguistics: ACL 2024
The growth of social media, characterized by its multimodal nature, has led to the emergence of diverse phenomena and challenges, which calls for an effective approach to uniformly solve automated tasks. The powerful Large Vision Language Models make it possible to handle a variety of tasks simultaneously, but even with carefully designed prompting methods, the general domain models often fall short in aligning with the unique speaking style and context of social media tasks. In this paper, we introduce a Large Vision Language Model for Social Media Processing (SoMeLVLM), which is a cognitive framework equipped with five key capabilities including knowledge & comprehension, application, analysis, evaluation, and creation. SoMeLVLM is designed to understand and generate realistic social media behavior. We have developed a 654k multimodal social media instruction-tuning dataset to support our cognitive framework and fine-tune our model. Our experiments demonstrate that SoMeLVLM achieves state-of-the-art performance in multiple social media tasks. Further analysis shows its significant advantages over baselines in terms of cognitive abilities.
Search
Fix author
Co-authors
- Jiebo Luo 6
- Nesreen K. Ahmed 2
- Ryan Aponte 2
- Hongjie Chen 2
- Franck Dernoncourt 2
- Zhengmian Hu 2
- Sungchul Kim 2
- Branislav Kveton 2
- Xintong Li 2
- Puneet Mathur 2
- Thien Nguyen 2
- Namyong Park 2
- Ryan A. Rossi 2
- Yu Wang 2
- Zhongyu Wei (魏忠钰) 2
- Junda Wu 2
- Yu Xia 2
- Zhouhang Xie 2
- Seunghyun Yoon 2
- Tong Yu 2
- Ruiyi Zhang 2
- Xinnong Zhang 2
- Joe Barrow 1
- Trung Bui 1
- Jian Chen 1
- Ren Chen 1
- Siming Chen 1
- Xiang Chen 1
- Hanieh Deilamsalehy 1
- Caiwen Gou 1
- Jiuxiang Gu 1
- Jinfa Huang 1
- Ting-Hao Huang 1
- Xuan-Jing Huang (黄萱菁) 1
- Song Jiang 1
- Jihyung Kil 1
- Haoyu Kuang 1
- Viet Dac Lai 1
- Chris Leung 1
- Nedim Lipka 1
- Julian McAuley 1
- Xinyi Mou (牟馨忆) 1
- Koyel Mukherjee 1
- Subhojyoti Mukherjee 1
- Dang Nguyen 1
- Soumyabrata Pal 1
- Jinsheng Pan 1
- Wanting Shan 1
- Jing Shi 1
- Jiajie Tang 1
- Mehrab Tanjim 1
- Zhongwei Wan 1
- Qifan Wang 1
- Shanshan Wang 1
- Zichao Wang 1
- Derek F. Wong (黄辉) 1
- Gang Wu 1
- Junchao Wu 1
- Kun Wu 1
- Yinglong Xia 1
- Cheng-zhong Xu 1
- Jingming Yao 1
- Lina Yao 1
- Fengying Ye 1
- Hanqing Zeng 1
- Si Zhang 1
- Zhehao Zhang 1
- Yue Zhao 1
- Tianyi Zhou 1