Lennart Kasserra
Author directory2026
Conjoint Auditing: A Framework for Measuring the Social Impact of LLM Deployment
Katrina M. Jones | Lennart Kasserra
Proceedings of the Fifth Workshop on NLP for Positive Impact (NLP4PI)
Katrina M. Jones | Lennart Kasserra
Proceedings of the Fifth Workshop on NLP for Positive Impact (NLP4PI)
How can we rigorously measure the social impact of LLMs in high-stakes deployments? Existing bias benchmarks flag problems at the model level but offer limited insight into how bias manifests across contexts, magnitudes, attributes, and their interactions. We propose conjoint auditing, a framework that adapts conjoint experimental design from market research to LLM bias evaluation. By presenting randomly generated multi-attribute profiles in a simulated deployment scenario, it delivers deployment-specific estimates of attribute effects and their interactions. Applying the framework for a financial analyst hiring scenario across two commercial LLMs (GPT-5.4 mini and Claude Haiku 4.5; roughly 40,000 observations each), we find that protected attributes exert small but statistically significant and substantively meaningful effects (1-2% of explained variance), with directions and magnitudes varying sharply by model. We also detect clear intersectional patterns, such as amplified disability penalties for women and reduced returns to experience for Black candidates.