Hidden Configuration Effects on Fairness in LLM-Based Multi-Agent Decision Systems

Hamed Loghmani, Ninareh Mehrabi, Homa Hosseinmardi, Amir Ghasemian


Abstract
Large language models (LLMs) deployed as decision agents in regulated settings—loan approval, hiring, clinical triage—must pass pre-deployment fairness audits that report a single fairness number per model under one implicit configuration. We argue this framing misses the dominant source of variation: configuration itself. Sweeping five axes (persona, interview, deliberation, aggregation rule, order) across three frontier LLMs (Claude Sonnet 4.6, GPT-4o-mini, Grok 4.20 Reasoning) on a 1,000-applicant Adult Census Income sample and a 500-applicant ACSIncome (2024 ACS data) sample, we find that within a single model, persona alone swings positive-prediction rate by up to 91 percentage points and collapses decision rules on two of three models; within a Conservative/Balanced/Progressive panel, replacing the panel chair on identical deliberation transcripts moves the disparate-impact ratio from adverse-impact to near-parity; and reversing the speaking order flips up to 34% of applicants on the same panel. We identify a median-voter effect: the Balanced agent matches the majority-vote verdict on 95.8–100% of applicants across all three models—deliberative aggregation reduces, mechanically, to single-agent decision by the median. The audit unit is the configuration, not the model: a system can pass an audit under one configuration and be silently reconfigured into a non-compliant one through a prompt edit—a change that current regulations do not treat as requiring re-audit. The code to reproduce the experiments in this paper is available at https://anonymous.4open.science/r/agent-decision-bias-emnlp-F059/
Anthology ID:
2026.emnlp-main.2531
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
57565–57589
Language:
URL:
https://aclanthology.org/2026.emnlp-main.2531/
DOI:
Bibkey:
Cite (ACL):
Hamed Loghmani, Ninareh Mehrabi, Homa Hosseinmardi, and Amir Ghasemian. 2026. Hidden Configuration Effects on Fairness in LLM-Based Multi-Agent Decision Systems. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 57565–57589, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Hidden Configuration Effects on Fairness in LLM-Based Multi-Agent Decision Systems (Loghmani et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.2531.pdf