Umaira Izhar
Author directory2026
Measuring Cultural Alignment Beyond the Average: A Framework for Evaluating Maternal-Health LLM Interactions in Indian Contexts
Umaira Izhar | Gunjan Arora | Pushpendra Singh
Findings of the Association for Computational Linguistics: EMNLP 2026
Umaira Izhar | Gunjan Arora | Pushpendra Singh
Findings of the Association for Computational Linguistics: EMNLP 2026
Existing evaluation methods for healthcare LLMs primarily assess factual correctness, safety, and fluency, while providing limited insight into whether generated interactions reflect culturally situated healthcare reasoning. This limitation is particularly important in maternal health, where care decisions are shaped by social and relational norms. We introduce MH-INDIC, a culturally grounded evaluation framework for maternal-health interactions in Indian contexts that operationalises cultural behaviour through ten dimensions of maternal-health reasoning. Using a 26-item survey administered to 102 pregnant and postpartum women from urban and semi-urban North India, we evaluate ten LLMs under profile-conditioned prompting. Our evaluation separates aggregate cultural alignment and context-dependent sensitivity to demographic and household factors. While several proprietary models approximate human population-level distributions, all evaluated systems exhibit substantially reduced behavioural variation across profiles. We further show that human-grounded conditioning enhances the realism and contextual coherence of synthetic maternal health dialogues. Together, our findings highlight the importance of evaluating healthcare LLMs not only for factual correctness but also for their ability to model socially situated reasoning in culturally embedded healthcare settings.