Fairness metrics such as demographic parity and equalized odds summarize group-level behavior, while the location of subgroup errors in the score distribution often remains unclear. RISE-NLP is a diagnostic interface for auditing probabilistic text classifiers through signed residual distributions. Given prediction scores, labels, group metadata, and text, RISE-NLP plots group residual-quantile curves, reports distributional metrics, and links selected percentile regions to the examples behind the gap. Using BERT and RoBERTa predictions, we observe F_dist=0.097--0.150 in the two identity-linked toxicity settings and 0.055--0.058 in a randomly grouped SST-2 control used as a descriptive reference. A 16-participant formative study evaluates users’ understanding and perceived usefulness of the residual visualizations, showing their potential to support reasoning about fairness–accuracy trade-offs. The project is available at https://ufdatastudio.com/projects/risenlp.