While AI toxicity detection models are widely used, their (un)reliability under realistic linguistic variation remains poorly understood. We provide a comprehensive study of how contextual change affects prediction reliability. We evaluate eight state-of-the-art toxicity detection models across three dimensions of linguistic variation: conversational context, conversational depth, and surface form transformations. Our results on three toxicity benchmarks show that conversational context often introduces substantial prediction instability, with some systems exhibiting prediction flip rates of up to 40%, while increasing conversational depth fails to produce consistent performance gains. Under paraphrasing transformations designed to preserve the underlying meaning and harmful intent, models exhibit substantial degradation and prediction flip rates of up to 57%. Overall, our findings show that strong benchmark performance does not necessarily translate to robust moderation behavior under realistic linguistic variation, highlighting the need for robustness-oriented model development and evaluation frameworks that better capture deployment conditions.