Deploying Large Language Models (LLMs) for content moderation on social media platforms introduces new challenges related to potential errors and biases, especially for speech-related policies. In this paper, we evaluate a diverse sample of 20 LLMs in zero-shot classification, for the specific case of sexist hate speech in French; a challenging task due to the scarcity of corpora in French and the English-centric research. We elaborate multiple protocols to explore the models’ understanding of the task and sensitivity to the language of the prompts and class labels. We show that small or medium size models can outperform much larger models, supporting the paradigm that scale is not the main driver of performance. We also demonstrate that models’ performance is not significantly altered when receiving specific guidelines about how to annotate potentially sexist content in French. Qualitative error analysis reveals that guidelines can help capture edge-cases (e.g. condemn sexism), but those are off-set by new misclassifications. This highlights a limitation of zero-shot prompting for content moderation, where specific instructions are crucial for adapting to different languages and cultures.