Textual anomaly detection (AD) aims to identify samples that deviate from the underlying distribution of normal data, without requiring explicit supervision on anomalies. Existing literature has focused almost exclusively on the semi-supervised setting, assuming access to clean training datasets composed solely of normal samples. However, this assumption rarely holds in real-world scenarios, as datasets used to infer the normal distribution are frequently contaminated by anomalies. In this work, we first highlight the sensitivity of conventional anomaly detection methods to contamination, before demonstrating the relevance of data depth functions for robust textual anomaly detection. Building on recent findings regarding the geometry of language model embeddings, further supported by our empirical observations, we introduce the geodesic depth to better capture the structure of contextual embedding spaces. Our results show that the geodesic depth consistently outperforms commonly used anomaly detection methods once contamination exceeds 10% in the training data, with substantially larger improvements observed in the most challenging scenarios.

Anomaly detection aims at distinguishing between in-distribution samples, which belong to the same distribution as the training set, and out-of-distribution samples, which lie outside of it. In textual anomaly detection, recent approaches routinely apply anomaly detection algorithms directly to embeddings extracted from pre-trained embedding models (two-stage approaches). However, the geometric properties of pre-trained embeddings can hinder the effectiveness of detection algorithms, which often rely on distance-based measures. In this work, we first highlight the relevance of similarity-trained models for textual anomaly detection. Beyond being trained to capture semantic similarities, these models also exhibit geometric properties that appear better suited to detection algorithms. We further demonstrate that, besides model choice, a simple post-processing step can significantly improve anomaly detection by adapting embeddings to the assumptions made by classical detection algorithms. The bulk of our experiments is done on a reformulation of the classification tasks from the MTEB benchmark into anomaly detection tasks.