Risa Ueno
2026
Reasoning Beyond Labels: Measuring LLM Sentiment in Low-Resource, Culturally Nuanced Contexts
Millicent Ochieng | Anja Thieme | Ignatius Ezeani | Risa Ueno | Samuel Chege Maina | Keshet Ronen | Javier Gonzalez | Jacki O'Neill
Proceedings of the 7th Workshop on African Natural Language Processing (AfricaNLP 2026)
Millicent Ochieng | Anja Thieme | Ignatius Ezeani | Risa Ueno | Samuel Chege Maina | Keshet Ronen | Javier Gonzalez | Jacki O'Neill
Proceedings of the 7th Workshop on African Natural Language Processing (AfricaNLP 2026)
Sentiment analysis in low-resource, culturally nuanced contexts challenges conventional NLP approaches that assume fixed labels and universal affective expressions. We present a diagnostic framework that treats sentiment as a context-dependent, culturally embedded construct, and evaluate how large language models (LLMs) reason about sentiment in informal, code-mixed WhatsApp messages from Nairobi youth health groups. Using human-annotated data, sentiment-flipped counterfactuals, and rubric-based explanation evaluation, we probe LLM interpretability, robustness, and alignment with human reasoning. Framing our evaluation through a social science measurement lens, we operationalize LLM outputs as an instrument for measuring the abstract concept of sentiment. Our findings reveal significant variation in model reasoning quality, with top-tier LLMs demonstrating greater interpretive stability, while smaller open-weight models in our study show reduced stability under ambiguity or sentiment shifts. This work highlights the need for culturally sensitive, reasoning-aware AI evaluation in complex, real-world communication.