Modern supervised fine-tuning (SFT) trains large language models on heterogeneous instruction mixtures, where the effective unit of optimization is an individual instruction-response pair rather than a coarse task label. Existing parameter isolation methods mitigate interference by assigning protected or trainable subspaces at the task or training stage level, but these units are often misaligned with actual parameter usage: instructions from the same task may require conflicting computations, while instructions from different tasks may share latent capabilities. We propose Instruction-Level Parameter Isolation (IPI), a fine-tuning framework that routes each instruction’s update to a sparse, capability-conditioned parameter subspace. IPI builds a multi-view instruction representation, predicts importance factors with a lightweight router, and uses an identity-trick reparameterization that preserves dense forward computation while exposing sparse base-model gradients and a task-loss saliency signal for router learning. Across heterogeneous instruction tuning benchmarks and multiple backbone models, IPI consistently improves performance over strong fine-tuning and parameter-isolation baselines. Further analyses show that instruction-conditioned masks reduce destructive gradient interactions and capture optimization structure that task labels miss.

Latent communication provides an efficient alternative to text-based exchange in multi-agent LLM systems by transmitting continuous hidden representations instead of natural-language messages. However, opacity is not privacy: latent messages may still carry private prompts, agent memory, or task-irrelevant sensitive information recoverable by downstream agents or adversaries. We study this risk through communicated latent messages, the edge-level representations that cross agent boundaries and define the privacy interface of latent multi-agent systems. We identify two coupled leakage mechanisms: instantaneous leakage from individual messages and propagation leakage accumulated across multi-round transcripts.

We propose Information-Constrained Latent Communication (ICLC), a model-agnostic communication-layer framework that constrains latent information exposure without modifying backbone LLM parameters. ICLC combines a receiver-conditioned latent bottleneck, adversarial regularization over messages and transcripts, and topology-aware gates that suppress unnecessary communication edges. Across private-context question answering, collaborative reasoning, and tool-use planning, ICLC preserves the utility benefit of latent communication while reducing average leakage from 0.747 to 0.398 compared with raw latent communication. These results suggest that privacy in latent multi-agent systems should be treated as a dynamic information-flow problem over messages, edges, and interaction rounds.

As AI assistants serve millions of users daily, evaluating user experience (UX) beyond general model capability has become increasingly important. We present UXBench, the first user-centric benchmark grounded in real user feedback signals for evaluating preference alignment and dialogue generation. The benchmark consists of three interconnected tasks, UX Judge, UX Eval, and UX Recovery, with 7,400 test instances extracted from over 70K interaction logs of a mainstream Chinese AI assistant. The dataset closely reflects real user distributions, covering 8 scenarios, 83 domains, and diverse failure patterns that pose severe challenges. Extensive experiments on 26 frontier language models provide novel insights into how well models perceive user experience and how improvements in model capability contribute to better dialogue engagement. Through analyses of model behavior and performance gaps, we document six important findings, demonstrating that user feedback prediction is a learnable capability and revealing different aspects that influence user experience. UXBench establishes a new evaluation landscape and calls for greater attention to tailored UX optimization, contributing toward a user-centric scaling law for the development of successful AI assistants. The full project is released at https://github.com/mengze-hong/UXBench.

Cementitious materials teams need machine-readable databases of compressive strength records to support mix-design optimization. However, manual curation of the literature is too slow and hard to scale. We present CemLit-IE, a five-stage zero-shot pipeline, converting raw PDFs into fully attributed records with provenance. The core challenge is that the relevant information is distributed across tables, text, and figures. Even though table-only extraction produces plausible numbers, the results are not auditable. CemLit-IE addresses this through a series of methods, including keyword-weighted page scoring, structured table injection, value-provenance schema, and review queue. On a benchmark of 20 papers with 655 expert-annotated records, CemLit-IE achieves a recall of 76.3% with 99.2% precision and 100% exact-match accuracy in a text-only model. The recall can be raised to 83.5% when figure reading is enabled. The ablation study shows that each design choice contributes to improving precision. As for the cost, a separate 270-paper journal-batch run extracted 2,039 strength records at $31.28 total API cost. All data and code are publicly released.

Cultural reasoning lies at the core of many emerging AI applications, yet existing benchmarks provide limited insight into how well models understand and contextualize cultural artifacts. We introduce ExhibitionBench, the first benchmark that frames exhibition curation as a high-level evaluation testbed, comprising 300 institution-derived exhibition and thematic collection records, 23,658 artifacts, and three interconnected tasks that evaluate complementary aspects of cultural understanding and reasoning, translating them into user-perceivable utilities. Extensive experiments across 29 systems reveal a significant gap between frontier LLMs and expert-level competence, particularly on tasks that require collection-level coherence and thematic understanding. Performance gains from scaling, prompting, and multimodal inputs are limited and highly model-dependent, whereas tailored model optimizations can equip open-weight models with better capabilities that support on-premise deployment. These findings highlight the difficulty of reasoning over cultural collections and provide a practical benchmark for developing more culturally grounded and reliable AI systems. The full project is released at https://github.com/mengze-hong/ExhibitionBench.

Maintaining persona consistency is paramount in the application of open-domain dialogue systems, as exemplified by models like ChatGPT. Despite significant advancements, the limited scale and diversity of current persona dialogue datasets remain challenges to achieving robust persona-consistent dialogue models. In this study, drawing inspiration from the success of large-scale pre-training, we introduce PPDS, an open-domain persona dialogue system that employs extensive generative pre-training on a persona dialogue dataset to enhance persona consistency. Specifically, we present a persona extraction model designed to autonomously and precisely generate vast persona dialogue datasets. Additionally, we unveil a pioneering persona augmentation technique to address the invalid persona bias inherent in the constructed dataset. Both quantitative and human evaluations consistently highlight the superior response quality and persona consistency of our proposed model, underscoring its effectiveness.

The rapid advancement of Chinese LLMs underscores the need for vertical-domain evaluations to ensure reliable applications. However, existing benchmarks often lack domain coverage and provide limited insights into the Chinese working context. Leveraging qualification exams as a unified framework for expertise evaluation, we introduce QualBench, the first multi-domain Chinese QA benchmark dedicated to localized assessment of Chinese LLMs. The dataset includes over 17,000 questions across six vertical domains, drawn from 24 Chinese qualifications to align with national policies and professional standards. Results reveal an interesting pattern of Chinese LLMs consistently surpassing non-Chinese models, with the Qwen2.5 model outperforming the more advanced GPT-4o, emphasizing the value of localized domain knowledge in meeting qualification requirements. The average accuracy of 53.98% reveals the current gaps in domain coverage within model capabilities. Furthermore, we identify performance degradation caused by LLM crowdsourcing, assess data contamination, and illustrate the effectiveness of prompt engineering and model fine-tuning, suggesting opportunities for future improvements through multi-domain RAG and Federated Learning. Data and code are publicly available at https://github.com/mengze-hong/QualBench.

Discovering customer intentions is crucial for automated service agents, yet existing intent clustering methods often fall short due to their reliance on embedding distance metrics and neglect of underlying semantic structures. To address these limitations, we propose an LLM-in-the-loop (LLM-ITL) intent clustering framework, integrating the language understanding capabilities of LLMs into conventional clustering algorithms. Specifically, this paper (1) examines the effectiveness of fine-tuned LLMs in semantic coherence evaluation and intent cluster naming, achieving over 95% accuracy aligned with human judgments; (2) designs an LLM-ITL framework that facilitates the iterative discovery of coherent intent clusters and the optimal number of clusters; and (3) introduces context-aware techniques tailored for customer service dialogue. Since existing English benchmarks lack sufficient semantic diversity and intent coverage, we further present a comprehensive Chinese dialogue intent dataset comprising over 100k real customer service calls with 1,507 human-annotated clusters. The proposed approaches significantly outperform LLM-guided baselines, achieving notable improvements in clustering quality, cost efficiency, and downstream applications. Combined with several best practices, our findings highlight the prominence of LLM-in-the-loop techniques for scalable dialogue data mining.

Retrieval-based chatbots leverage human-verified Q&A knowledge to deliver accurate, verifiable responses, making them ideal for customer-centric applications where compliance with regulatory and operational standards is critical. To effectively handle diverse customer inquiries, augmenting the knowledge base with “similar questions” that retain semantic meaning while incorporating varied expressions is a cost-effective strategy. In this paper, we introduce the Similar Question Generation (SQG) task for LLM training and inference, proposing context-aware approaches to enable comprehensive semantic exploration and enhanced alignment with source question-answer relationships. We formulate optimization techniques for constructing in-context prompts and selecting an optimal subset of similar questions to expand chatbot knowledge under budget constraints. Both quantitative and human evaluations validate the effectiveness of these methods, achieving a 92% user satisfaction rate in a deployed chatbot system, reflecting an 18% improvement over the unaugmented baseline. These findings highlight the practical benefits of SQG and emphasize the potential of LLMs, not as direct chatbot interfaces, but in supporting non-generative systems for hallucination-free, compliance-guaranteed applications.