Jiale Mao
Author directory2026
From Historical Queries to Actionable Rubrics: Category-Aware Rubric Generation for Local-Life AI Search
Jiale Mao | Fuwei Zhang | Dongbo Xi | Zhen Chen | Yifan Yang | Peng Yan | Zhao Zhang | Fuzhen Zhuang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Jiale Mao | Fuwei Zhang | Dongbo Xi | Zhen Chen | Yifan Yang | Peng Yan | Zhao Zhang | Fuzhen Zhuang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Rubric generation improves the interpretability and actionability of large language model evaluation, but in online-to-offline (O2O) local-life AI search, user queries often reflect more complex needs, such as consumption intent, location constraints, and scenario preferences. Directly generating rubrics from a single query tends to produce generic criteria that fail to satisfy AI search evaluation needs, while leveraging candidate responses, preference samples, or external evidence at scale remains challenging. To address these limitations, we propose Category-Aware Rubric Generation (CaRG), a category-aware evaluation dimension candidate pool framework. Through embedding-based clustering, representative selection, and iterative LLM-based consolidation, CaRG constructs reusable candidate pools of both general and category-specific evaluation dimensions from representative historical queries. Subsequently, these candidate pools are leveraged to generate query-level rubrics with concrete positive and negative scoring criteria for target queries. Experimental results demonstrate that CaRG consistently outperforms Direct-Rubric and shows more stable overall quality than existing query-only baselines, while achieving the strongest overall agreement with production golden rubric rankings. Our repository is available at https://github.com/PlussLe/CaRG.