QiYuan Shi
Author directory2026
Beyond Single-Attribute Auditing: A Configurable Multi-Attribute Benchmark for Intersectional Fairness in LLMs
Zheng Zhang | Yifan Wu | Yuxiang Zhou | Qi Liu | Qingyang Mao | Ning Li | Weibo Gao | QiYuan Shi | Zhenya Huang | Baosheng Yu
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Zheng Zhang | Yifan Wu | Yuxiang Zhou | Qi Liu | Qingyang Mao | Ning Li | Weibo Gao | QiYuan Shi | Zhenya Huang | Baosheng Yu
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Large language models (LLMs) are increasingly evaluated for fairness, yet most existing benchmarks audit one sensitive attribute at a time. This single-attribute paradigm can miss intersectional harms, where multiple attributes interact non-additively and produce failures that are invisible under marginal evaluation. We introduce ComFair, a configurable benchmark that converts existing fairness data into 479,066 controlled Dual- and Multi-attribute instances and pairs every augmented instance with its task-equivalent source. Across Recognition, Selection, Continuation, and Conversation, this pairing supports a unified audit of performance or bias-score shifts, how often added attributes introduce new errors or mask existing ones, and model-ranking stability. Experiments on ten LLMs show task- and subgroup-dependent sensitivity: discriminative tasks expose both transition directions even when aggregate changes are modest, generative tasks exhibit larger and directionally asymmetric shifts, and rankings vary with attribute composition. ComFair therefore provides a scalable controlled sensitivity audit—rather than a model of the natural joint distribution of identities—for testing the robustness of fairness-benchmark conclusions beyond single-attribute settings.