Jiaxuan You
Author directory2026
GMTRouter: Personalized LLM Router over Multi-turn User Interactions
Yihang Sun | Encheng Xie | Tao Feng | Jiaxuan You
Findings of the Association for Computational Linguistics: EMNLP 2026
Yihang Sun | Encheng Xie | Tao Feng | Jiaxuan You
Findings of the Association for Computational Linguistics: EMNLP 2026
Large Language Model (LLM) routing has demonstrated strong capability in balancing response quality with computational cost. As users exhibit diverse preferences, personalization has attracted increasing attention in LLM routing, since even identical queries may require different models to generate responses tailored to individual needs. However, existing approaches are not fully personalized and often fail to faithfully capture the complex interactions between users and LLMs. Moreover, user preference data is typically scarce and inconsistent in format, which limits the effectiveness of methods that directly leverage user-specific data. To address these challenges, we propose GMTRouter, which represents multi-turn user–LLM interactions as a heterogeneous graph with five node types: user, LLM, query, response and turn, thereby maximally preserving the rich relational structure of the interaction. Through a lightweight inductive graph learning framework combined with a tailored user-conditioned graph sampling mechanism, GMTRouter learns to capture user preferences from few-shot data, enabling effective personalization. Extensive experiments demonstrate that GMTRouter outperforms the strongest baselines, achieving up to a 0.108 absolute improvement in accuracy and a 0.124 improvement in AUC. More importantly, we further demonstrate that GMTRouter can adapt to new users using only few-shot data, without extensive fine-tuning.
CUADebug: Diagnosing and Repairing Computer-Use Agent Failures
Weijia Zhang | Kunlun Zhu | Zeyi Liu | Yinting Chen | Tianyi Ma | Jiateng Liu | Jiaxun Zhang | Bingxuan Li | Pan Lu | Xiangru Tang | Heng Ji | Jiaxuan You
Findings of the Association for Computational Linguistics: EMNLP 2026
Weijia Zhang | Kunlun Zhu | Zeyi Liu | Yinting Chen | Tianyi Ma | Jiateng Liu | Jiaxun Zhang | Bingxuan Li | Pan Lu | Xiangru Tang | Heng Ji | Jiaxuan You
Findings of the Association for Computational Linguistics: EMNLP 2026
Computer-use agents (CUAs) interact with graphical interfaces through screenshots and low-level mouse and keyboard actions, yet the causal error may precede the terminal failure. We present **CUADebug**, a framework for localizing root causes in CUA trajectories and guiding re-execution. **CUADebug** includes a five-category, 30-subtype taxonomy; CUAErrorBench, a benchmark of 204 failed OSWorld trajectories with human root-cause annotations; and CUADebugger, a ReAct-style agent for root-cause analysis (RCA). CUADebugger iteratively selects trajectory steps, inspects paired before/after screenshots and action traces, and submits a structured diagnosis containing the causal step, taxonomy label, grounded evidence, and correction. CUADebugger performs RCA without per-trajectory human intervention; human annotations are used to evaluate RCA predictions and, in controlled re-rollout comparisons, to fix restart points. Task reasoning and control is the largest annotated failure category (110/204). CUADebugger improves L2 and Tag+Step Exact across three debugger backbones on the Claude-agent split; with Gemini 2.5 Pro, Tag+Step Exact rises from 11.1% to 19.4%. Single re-execution improves failure recovery from 13.89% to 29.86% (overall 61.77% to 68.14%); controlled continual re-execution improves it from 12.50% to 25.69% (overall 61.22% to 66.48%). Project page: cuadebug.github.io
Can Large Language Models Uncover the Structure of Social Opinions?
Chongrui Ye | Yuxiang Liu | Haofei Yu | Jiayu Liu | Jiaxuan You
Findings of the Association for Computational Linguistics: EMNLP 2026
Chongrui Ye | Yuxiang Liu | Haofei Yu | Jiayu Liu | Jiaxuan You
Findings of the Association for Computational Linguistics: EMNLP 2026
Understanding how opinions on different issues evolve together is essential for modeling collective intelligence, yet this remains underexplored due to the absence of standardized benchmarks. We introduce the concept of an opinion graph, where nodes represent social opinions on real-world events (e.g., presidential elections, stock predictions) and edges capture pairwise relationships between them. Building on this, we present OpinionBench, a continuously evolving benchmark derived from multiple live prediction markets, designed to evaluate whether large language models (LLMs) can uncover the latent structure within evolving social opinions. OpinionBench employs an scalable and automated graph construction and labeling pipeline, which assigns edge labels using time-series co-movement, semantic similarity, and metadata, followed by human validation. Experiments show that (1) LLMs consistently outperform baselines in identifying opinion correlations across domains, and (2) LLMs can recover the underlying graph structure through edge prediction. OpinionBench provides a challenging testbed for assessing LLMs’ ability to capture complex patterns of social opinion co-evolution.
DRPG (Decompose, Retrieve, Plan, Generate): An Agentic Framework for Academic Rebuttal
Peixuan Han | YingJie Yu | Jingjun Xu | Jiaxuan You
Findings of the Association for Computational Linguistics: ACL 2026
Peixuan Han | YingJie Yu | Jingjun Xu | Jiaxuan You
Findings of the Association for Computational Linguistics: ACL 2026
Despite the growing adoption of large language models (LLMs) in scientific research workflows, automated support for academic rebuttal, a crucial step in academic communication and peer review, remains largely underexplored. Existing approaches typically rely on off-the-shelf LLMs or simple pipelines, which struggle with long-context understanding and often fail to produce targeted and persuasive responses. In this paper, we propose DRPG, an agentic framework for automatic academic rebuttal generation that operates through four steps: Decompose reviews into atomic concerns, Retrieve relevant evidence from the paper, Plan rebuttal strategies, and Generate responses accordingly. Notably, the Planner in DRPG reaches over 98% accuracy in identifying the most feasible rebuttal direction. Experiments on data from top-tier conferences demonstrate that DRPG significantly outperforms existing rebuttal pipelines and achieves performance beyond the average human level using only an 8B model. Our analysis further demonstrates the effectiveness of the planner design and its value in providing multi-perspective and explainable suggestions. We also showed that DRPG works well in a more complex multi-round setting. These results highlight the effectiveness of DRPG and its potential to provide high-quality rebuttal content and support the scaling of academic discussions.
Can Large Language Models Forecast What Researchers Study Next?
Fenghai Li | Zihan Tang | Haofei Yu | Yining Zhao | Jiaxuan You
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Fenghai Li | Zihan Tang | Haofei Yu | Yining Zhao | Jiaxuan You
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Large language models increasingly generate research ideas, yet judging their novelty or feasibility at generation time does not establish whether they anticipate subsequent work. We introduce IdeaForecastBench to evaluate research idea forecasting. Given a community’s literature up to a cutoff, a system produces up to five ranked ideas, which are evaluated against later papers. The benchmark comprises 624 rolling episodes across 52 topics, with a fixed retrieve-then-judge protocol and separately reported results from two judges. We compare five history-compression strategies across GPT-4.1, Qwen2.5-7B/14B, and Qwen3.5-9B, together with a learned Mode-Decomposition Forecaster (MDF). Under the primary GPT-4.1-mini judge, Summary improves on Direct in Hit@5 and Precision@5 across all four backbones. Qwen2.5 scores above GPT-4.1, whereas Qwen3.5 scores below it. An outcome-blind assessment finds that Qwen2.5 produces broader forecasts, but does not identify how much breadth contributes to its advantage. Threshold and judge diagnostics further clarify the limits of interpreting realization as precise anticipation. IdeaForecastBench provides a common task for studying which research ideas a community subsequently pursues and how reliably this outcome can be measured.
Multi-Agent Evolve: LLM Self-Improve through Co-evolution
Yixing Chen | Yiding Wang | Siqi Zhu | Weixian Xu | Haofei Yu | Tao Feng | Muhan Zhang | Mostofa Patwary | Jiaxuan You
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Yixing Chen | Yiding Wang | Siqi Zhu | Weixian Xu | Haofei Yu | Tao Feng | Muhan Zhang | Mostofa Patwary | Jiaxuan You
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Reinforcement Learning (RL) has demonstrated significant potential in enhancing the capabilities of large language models (LLMs). However, the success of RL for LLMs heavily relies on human-curated datasets, which pose a burden on scalability and sustainability. Inspired by the success of the paradigm in games and Go, Self-Play methods are deployed to enhance LLM capabilities without human-annotated data. However, these methods primarily depend on a grounded environment for verifiable feedback (e.g., a Python interpreter or a game engine); extending them to general domains remains challenging. To address these challenges, we propose **Multi-Agent Evolve (MAE)**, a framework in which LLM can self-evolve across diverse tasks from mathematics to general knowledge. MAE implements asymmetric Self-Play by leveraging a triplet of interacting roles (*Proposer, Solver, Judge*) that are instantiated from a single LLM. *Proposer* generates challenging questions, *Solver* optimizes solutions, and *Judge* provides reward signals. These roles are jointly optimized via reinforcement learning, creating a closed-loop system where the model continuously challenges and improves itself. Experiments on Qwen2.5-3B-Instruct demonstrate that MAE achieves an average improvement of 4.86% across multiple benchmarks, significantly outperforming baselines. These results highlight MAE as a scalable, data-efficient method for enhancing LLMs’ general reasoning abilities with minimal reliance on human supervision.
AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents
Kunlun Zhu | Xuyan Ye | Zhiguang Han | Yuchen Zhao | Bingxuan Li | Weijia Zhang | Muxin Tian | Xiangru Tang | Pan Lu | James Zou | Jiaxuan You | Heng Ji
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
Kunlun Zhu | Xuyan Ye | Zhiguang Han | Yuchen Zhao | Bingxuan Li | Weijia Zhang | Muxin Tian | Xiangru Tang | Pan Lu | James Zou | Jiaxuan You | Heng Ji
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
LLM agent failures are difficult to debug because the step where an error surfaces is often not the one that caused it. Existing observability tools replay execution traces but provide little support for identifying the root cause or translating diagnosis into recovery. We present AgentDebugX, an open-source debugging framework that organizes debugging as a closed loop of Detect, Attribute, Recover, and Rerun. At its core, DeepDebug performs multi-turn root-cause diagnosis through global trajectory understanding, structure-guided investigation, and cross-examination. On the Who When benchmark, DeepDebug achieves the best strict attribution accuracy among the evaluated methods on both tested open-weight backbones, reaching 28.8% exact agent-and-step accuracy on qwen3.5-9b, compared with 21.7% for the strongest single-pass baseline. On GAIA, DeepDebug repairs 13 of 73 failed tasks in a single rerun, compared with 4–6 for three decoupled self-correction baselines, improving overall accuracy from 55.8% to 63.6%. AgentDebugX exposes this workflow through a Python library, CLI, web console, and installable agentic skill. It also provides an opt-in Error Hub for sharing scrubbed failure–diagnosis–repair bundles and reusing them as debugging memory.
GUIDE: Towards Scalable Advising for Research Ideas
Yaowenqi Liu | Bingxu Meng | Rui Pan | Yuxing Liu | Jerry Huang | Jiaxuan You | Tong Zhang
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Yaowenqi Liu | Bingxu Meng | Rui Pan | Yuxing Liu | Jerry Huang | Jiaxuan You | Tong Zhang
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
The field of AI research is advancing at an unprecedented pace, enabling automated hypothesis generation and experimental design across diverse domains such as biology, mathematics, and artificial intelligence. Despite these advancements, there remains a significant gap in the availability of scalable advising systems capable of providing high-quality, well-reasoned feedback to refine proposed hypotheses and experimental designs. To address this challenge, we explore key factors that underlie the development of robust advising systems, including model size, data reweighting, context length, confidence estimation, and structured reasoning processes. Our findings reveal that a relatively small model, when equipped with a well-compressed literature database and a structured reasoning framework, can outperform powerful general-purpose language models such as Deepseek-R1 in terms of acceptance rates for self-ranked top-30% submissions to ICLR 2025. Moreover, when limited to high-confidence predictions, our system achieves an acceptance rate exceeding 90% on the ICLR 2025 test set, underscoring its potential to significantly enhance the quality and efficiency of hypothesis generation and experimental design.
2025
Tables as Thought: Exploring Structured Thoughts in LLM Reasoning
Zhenjie Sun | Naihao Deng | Haofei Yu | Jiaxuan You
Proceedings of the 4th Table Representation Learning Workshop
Zhenjie Sun | Naihao Deng | Haofei Yu | Jiaxuan You
Proceedings of the 4th Table Representation Learning Workshop
Large language models’ reasoning abilities benefit from methods that organize their thought processes, such as chain-of-thought prompting, which employs a sequential structure to guide the reasoning process step-by-step. However, existing approaches focus primarily on organizing the sequence of thoughts, leaving structure in individual thought steps underexplored. To address this gap, we propose Table as Thought, a framework inspired by cognitive neuroscience theories on human thought. Table as Thought organizes reasoning within a tabular schema, where rows represent sequential thought steps and columns capture critical constraints and contextual information to enhance reasoning. The reasoning process iteratively populates the table until self-verification ensures completeness and correctness. Our experiments show that Table as Thought excels in planning tasks and demonstrates a strong potential for enhancing LLM performance in mathematical reasoning compared to unstructured thought baselines. This work provides a novel exploration of refining thought representation within LLMs, paving the way for advancements in reasoning and AI cognition.
Evaluating LLM-Generated Diagrams as Graphs
Chumeng Liang | Jiaxuan You
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Chumeng Liang | Jiaxuan You
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Diagrams play a central role in research papers for conveying ideas, yet they are often notoriously complex and labor-intensive to create. Although diagrams are presented as images, standard image generative models struggle to produce clear diagrams with well-defined structure. We argue that a promising direction is to generate demonstration diagrams directly in textual form as SVGs, which can leverage recent advances in large language models (LLMs). However, due to the complexity of components and the multimodal nature of diagrams, sufficiently discriminative and explainable metrics for evaluating the quality of LLM-generated diagrams remain lacking. In this paper, we propose DiagramEval, a novel evaluation metric designed to assess demonstration diagrams generated by LLMs. Specifically, DiagramEval conceptualizes diagrams as graphs, treating text elements as nodes and their connections as directed edges, and evaluates diagram quality using two new groups of metrics: node alignment and path alignment. For the first time, we effectively evaluate diagrams produced by state-of-the-art LLMs on recent research literature, quantitatively demonstrating the validity of our metrics. Furthermore, we show how the enhanced explainability of our proposed metrics offers valuable insights into the characteristics of LLM-generated diagrams.
SafeScientist: Enhancing AI Scientist Safety for Risk-Aware Scientific Discovery
Kunlun Zhu | Jiaxun Zhang | Ziheng Qi | Nuoxing Shang | Zijia Liu | Peixuan Han | Yue Su | Haofei Yu | Jiaxuan You
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Kunlun Zhu | Jiaxun Zhang | Ziheng Qi | Nuoxing Shang | Zijia Liu | Peixuan Han | Yue Su | Haofei Yu | Jiaxuan You
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Recent advancements in large language model (LLM) agents have significantly accelerated scientific discovery automation, yet concurrently raised critical ethical and safety concerns. To systematically address these challenges, we introduce SafeScientist, an innovative AI scientist framework explicitly designed to enhance safety and ethical responsibility in AI-driven scientific exploration. SafeScientist proactively refuses ethically inappropriate or high-risk tasks and rigorously emphasizes safety throughout the research process. To achieve comprehensive safety oversight, we integrate multiple defensive mechanisms, including prompt monitoring, agent-collaboration monitoring, tool-use monitoring, and an ethical reviewer component. Complementing SafeScientist, we propose SciSafetyBench , a novel benchmark specifically designed to evaluate AI safety in scientific contexts, comprising 240 high-risk scientific tasks across 6 domains, alongside 30 specially designed scientific tools and 120 tool-related risk tasks. Extensive experiments demonstrate that SafeScientist significantly improves safety performance by 35% compared to traditional AI scientist frameworks, without compromising scientific output quality. Additionally, we rigorously validate the robustness of our safety pipeline against diverse adversarial attack methods, further confirming the effectiveness of our integrated approach. The code and data will be available at https://github.com/ulab-uiuc/SafeScientist. Warning: this paper contains example data that may be offensive or harmful.
TinyScientist: An Interactive, Extensible, and Controllable Framework for Building Research Agents
Haofei Yu | Keyang Xuan | Fenghai Li | Kunlun Zhu | Zijie Lei | Jiaxun Zhang | Ziheng Qi | Kyle Richardson | Jiaxuan You
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
Haofei Yu | Keyang Xuan | Fenghai Li | Kunlun Zhu | Zijie Lei | Jiaxun Zhang | Ziheng Qi | Kyle Richardson | Jiaxuan You
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
Automatic research with Large Language Models (LLMs) is rapidly gaining importance, driving the development of increasingly complex workflows involving multi-agent systems, planning, tool usage, code execution, and human-agent interaction to accelerate research processes. However, as more researchers and developers begin to use and build upon these tools and platforms, the complexity and difficulty of extending and maintaining such agentic workflows have become a significant challenge, particularly as algorithms and architectures continue to advance. To address this growing complexity, TinyScientist identifies the essential components of the automatic research workflow and proposes an interactive, extensible, and controllable framework that adapts easily to new tools and supports iterative growth. We provide an open-source codebase, an interactive web demonstration, and a PyPI Python package to make state-of-the-art auto-research pipelines broadly accessible to every researcher and developer.
MultiAgentBench : Evaluating the Collaboration and Competition of LLM agents
Kunlun Zhu | Hongyi Du | Zhaochen Hong | Xiaocheng Yang | Shuyi Guo | Zhe Wang | Zhenhailong Wang | Cheng Qian | Xiangru Tang | Heng Ji | Jiaxuan You
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Kunlun Zhu | Hongyi Du | Zhaochen Hong | Xiaocheng Yang | Shuyi Guo | Zhe Wang | Zhenhailong Wang | Cheng Qian | Xiangru Tang | Heng Ji | Jiaxuan You
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Large Language Models (LLMs) have shown remarkable capabilities as autonomous agents; yet existing benchmarks either focus on single-agent tasks or are confined to narrow domains, failing to capture the dynamics of multi-agent coordination and competition. In this paper, we introduce MultiAgentBench, a comprehensive benchmark designed to evaluate LLM-based multi-agent systems across diverse, interactive scenarios. Our framework measures not only task completion but also the quality of collaboration and competition using novel, milestone-based key performance indicators. Moreover, we evaluate various coordination protocols (including star, chain, tree, and graph topologies) and innovative strategies such as group discussion and cognitive planning. Notably, gpt-4o-mini reaches the average highest task score, graph structure performs the best among coordination protocols in the research scenario,and cognitive planning improves milestone achievement rates by 3%. Code and datasets are publicavailable at https://github.com/ulab-uiuc/MARBLE.
Beyond Facts: Evaluating Intent Hallucination in Large Language Models
Yijie Hao | Haofei Yu | Jiaxuan You
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Yijie Hao | Haofei Yu | Jiaxuan You
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
When exposed to complex queries containing multiple conditions, today’s large language models (LLMs) tend to produce responses that only partially satisfy the query while neglecting certain conditions. We, therefore, introduce the concept of Intent Hallucination, a phenomenon where LLMs either omit (failing to address certain parts) or misinterpret (responding to invented query parts) elements of the given query, leading to responses misaligned with the original query. To systematically evaluate intent hallucination, we introduce FAITHQA, a novel benchmark for intent hallucination that contains 20,068 problems, covering both query-only and retrieval-augmented generation (RAG) setups with varying topics and difficulty. FAITHQA is the first hallucination benchmark that goes beyond factual verification, tailored to identify the fundamental cause of intent hallucination. By evaluating various LLMs on FAITHQA, we find that (1) intent hallucination is a common issue even for state-of-the-art models, and (2) such a phenomenon stems from omission or misinterpretation of LLMs. To facilitate future research, we introduce an automatic LLM generation evaluation metric, named INTENT CONSTRAINT, for detecting intent hallucination. Human evaluation results demonstrate that INTENT CONSTRAINT is closer to human performance for intent hallucination compared to baselines.
ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities
Zhaochen Hong | Haofei Yu | Jiaxuan You
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Zhaochen Hong | Haofei Yu | Jiaxuan You
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Evaluating Large Language Models (LLMs) requires effective methods to assess semantic consistency across multiple reversible transformations. Traditional self-consistency methods often fail to capture subtle semantic errors in multi-step tasks. We introduce ConsistencyChecker, a tree-based evaluation framework that measures LLMs’ ability to preserve semantic consistency during reversible transformation processes, sidestepping benchmark data contamination issues. Our approach constructs self-consistency trees where nodes represent text states after transformations (e.g., translation, code modification, paraphrasing) and edges represent pairs of opposite transformations. By analyzing semantic preservation between nodes at different tree depths, ConsistencyChecker quantifies model reliability without requiring manually annotated reference data. Experiments demonstrate that ConsistencyChecker reliably measures generalization abilities across models from 1.5B to 72B parameters. On translation tasks, GPT-4o Mini achieves the highest L3 consistency score of 98.0%. For code generation, Qwen 2.5 32B leads with 85.1% semantic consistency at L3. Results show Pearson correlation greater than 0.7 between our embedding-based scores and WMT 2024 rankings on 4 out of 5 shared language pairs, validating the method’s effectiveness for benchmarking LLM performance without constructing new datasets.
Graph of Records: Boosting Retrieval Augmented Generation for Long-context Summarization with Graphs
Haozhen Zhang | Tao Feng | Jiaxuan You
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Haozhen Zhang | Tao Feng | Jiaxuan You
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Retrieval-augmented generation (RAG) has revitalized Large Language Models (LLMs) by injecting non-parametric factual knowledge. Compared with long-context LLMs, RAG is considered an effective summarization tool in a more concise and lightweight manner, which can interact with LLMs multiple times using diverse queries to get comprehensive responses. However, the LLM-generated historical responses, which contain potentially insightful information, are largely neglected and discarded by existing approaches, leading to suboptimal results. In this paper, we propose graph of records (GoR), which leverages historical responses generated by LLMs to enhance RAG for long-context global summarization. Inspired by the retrieve-then-generate paradigm of RAG, we construct a graph by establishing an edge between the retrieved text chunks and the corresponding LLM-generated response. To further uncover the intricate correlations between them, GoR features a graph neural network and an elaborately designed BERTScore-based objective for self-supervised model training, enabling seamless supervision signal backpropagation between reference summaries and node embeddings. We comprehensively compare GoR with 12 baselines across four long-context summarization datasets, and the results indicate that our proposed method reaches the best performance (e.g., 15%, 8%, and 19% improvement over retrievers w.r.t. Rouge-L, Rouge-1, and Rouge-2 on the WCEP dataset). Extensive experiments further demonstrate the effectiveness of GoR.
2024
In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained Language Models
Pengrui Han | Peiyang Song | Haofei Yu | Jiaxuan You
Findings of the Association for Computational Linguistics: EMNLP 2024
Pengrui Han | Peiyang Song | Haofei Yu | Jiaxuan You
Findings of the Association for Computational Linguistics: EMNLP 2024
Recent advancements in artificial intelligence have led to the creation of highly capable large language models (LLMs) that can perform tasks in a human-like manner. However, LLMs exhibit only infant-level cognitive abilities in certain areas. One such area is the A-Not-B error, a phenomenon seen in infants where they repeat a previously rewarded behavior despite well-observed changed conditions. This highlights their lack of inhibitory control – the ability to stop a habitual or impulsive response. In our work, we design a text-based multi-choice QA scenario similar to the A-Not-B experimental settings to systematically test the inhibitory control abilities of LLMs. We found that state-of-the-art LLMs (like Llama3-8b) perform consistently well with in-context learning (ICL) but make errors and show a significant drop of as many as 83.3% in reasoning tasks when the context changes trivially. This suggests that LLMs only have inhibitory control abilities on par with human infants in this regard, often failing to suppress the previously established response pattern during ICL.
LLM-Evolve: Evaluation for LLM’s Evolving Capability on Benchmarks
Jiaxuan You | Mingjie Liu | Shrimai Prabhumoye | Mostofa Patwary | Mohammad Shoeybi | Bryan Catanzaro
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Jiaxuan You | Mingjie Liu | Shrimai Prabhumoye | Mostofa Patwary | Mohammad Shoeybi | Bryan Catanzaro
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
The advancement of large language models (LLMs) has extended their use to dynamic and interactive real-world applications, where models engage continuously with their environment and potentially enhance their performance over time. Most existing LLM benchmarks evaluate LLMs on i.i.d. tasks, overlooking their ability to learn iteratively from past experiences. Our paper bridges this evaluation gap by proposing a novel framework, LLM-Evolve, which extends established benchmarks to sequential problem-solving settings. LLM-Evolve evaluates LLMs over multiple rounds, providing feedback after each round to build a demonstration memory that the models can query in future tasks. We applied LLM-Evolve to the MMLU, GSM8K, and AgentBench benchmarks, testing 8 state-of-the-art open-source and closed-source models. Results show that LLMs can achieve performance improvements of up to 17% by learning from past interactions, with the quality of retrieval algorithms and feedback significantly influencing this capability. These insights advocate for more understanding and benchmarks for LLMs’ performance in evolving interactive scenarios.
Arxiv Copilot: A Self-Evolving and Efficient LLM System for Personalized Academic Assistance
Guanyu Lin | Tao Feng | Pengrui Han | Ge Liu | Jiaxuan You
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
Guanyu Lin | Tao Feng | Pengrui Han | Ge Liu | Jiaxuan You
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
As scientific research proliferates, researchers face the daunting task of navigating and reading vast amounts of literature. Existing solutions, such as document QA, fail to provide personalized and up-to-date information efficiently. We present Arxiv Copilot, a self-evolving, efficient LLM system designed to assist researchers, based on thought-retrieval, user profile and high performance optimization. Specifically, Arxiv Copilot can offer personalized research services, maintaining a real-time updated database. Quantitative evaluation demonstrates that Arxiv Copilot saves 69.92% of time after efficient deployment. This paper details the design and implementation of Arxiv Copilot, highlighting its contributions to personalized academic support and its potential to streamline the research process. We have deployed Arxiv Copilot at: https://huggingface.co/spaces/ulab-ai/ArxivCopilot.
Search
Co-authors
- Haofei Yu 9
- Kunlun Zhu 5
- Tao Feng 4
- Heng Ji 3
- Xiangru Tang 3
- Jiaxun Zhang 3
- Peixuan Han 2
- Pengrui Han 2
- Zhaochen Hong 2
- Bingxuan Li 2
- Fenghai Li 2
- Pan Lu 2
- Mostofa Patwary 2
- Ziheng Qi 2
- Weijia Zhang 2
- Bryan Catanzaro 1
- Yinting Chen 1
- Yixing Chen 1
- Naihao Deng 1
- Hongyi Du 1
- Shuyi Guo 1
- Zhiguang Han 1
- Yijie Hao 1
- Jerry Huang 1
- Zijie Lei 1
- Chumeng Liang 1
- Guanyu Lin 1
- Ge Liu 1
- Jiateng Liu 1
- Jiayu Liu 1
- Mingjie Liu 1
- Yaowenqi Liu 1
- Yuxiang Liu 1
- Yuxing Liu 1
- Zeyi Liu 1
- Zijia Liu 1
- Tianyi Ma 1
- Bingxu Meng 1
- Rui Pan 1
- Shrimai Prabhumoye 1
- Cheng Qian 1
- Kyle Richardson 1
- Nuoxing Shang 1
- Mohammad Shoeybi 1
- Peiyang Song 1
- Yue Su 1
- Yihang Sun 1
- Zhenjie Sun 1
- Zihan Tang 1
- Muxin Tian 1
- Yiding Wang 1
- Zhe Wang 1
- Zhenhailong Wang 1
- Encheng Xie 1
- Jingjun Xu 1
- Weixian Xu 1
- Keyang Xuan 1
- Xiaocheng Yang 1
- Chongrui Ye 1
- Xuyan Ye 1
- YingJie Yu 1
- Haozhen Zhang 1
- Muhan Zhang 1
- Tong Zhang 1
- Yining Zhao 1
- Yuchen Zhao 1
- Siqi Zhu 1
- James Zou 1