RISE-NLP: Interactive Residual Distribution Diagnostics for Fairness Auditing in Text Classification

Ray Chen, Kevin Kurian, Christan Grant


Abstract
Fairness metrics such as demographic parity and equalized odds summarize group-level behavior, while the location of subgroup errors in the score distribution often remains unclear. RISE-NLP is a diagnostic interface for auditing probabilistic text classifiers through signed residual distributions. Given prediction scores, labels, group metadata, and text, RISE-NLP plots group residual-quantile curves, reports distributional metrics, and links selected percentile regions to the examples behind the gap. Using BERT and RoBERTa predictions, we observe Fdist=0.097--0.150 in the two identity-linked toxicity settings and 0.055--0.058 in a randomly grouped SST-2 control used as a descriptive reference. A 16-participant formative study evaluates users’ understanding and perceived usefulness of the residual visualizations, showing their potential to support reasoning about fairness–accuracy trade-offs. The project is available at https://ufdatastudio.com/projects/risenlp.
Anthology ID:
2026.emnlp-demos.11
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Sweta Agrawal, Alberto Abad
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
120–127
Language:
URL:
https://aclanthology.org/2026.emnlp-demos.11/
DOI:
Bibkey:
Cite (ACL):
Ray Chen, Kevin Kurian, and Christan Grant. 2026. RISE-NLP: Interactive Residual Distribution Diagnostics for Fairness Auditing in Text Classification. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 120–127, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
RISE-NLP: Interactive Residual Distribution Diagnostics for Fairness Auditing in Text Classification (Chen et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-demos.11.pdf