Pushpendra Singh
Author directory2026
Measuring Cultural Alignment Beyond the Average: A Framework for Evaluating Maternal-Health LLM Interactions in Indian Contexts
Umaira Izhar | Gunjan Arora | Pushpendra Singh
Findings of the Association for Computational Linguistics: EMNLP 2026
Umaira Izhar | Gunjan Arora | Pushpendra Singh
Findings of the Association for Computational Linguistics: EMNLP 2026
Existing evaluation methods for healthcare LLMs primarily assess factual correctness, safety, and fluency, while providing limited insight into whether generated interactions reflect culturally situated healthcare reasoning. This limitation is particularly important in maternal health, where care decisions are shaped by social and relational norms. We introduce MH-INDIC, a culturally grounded evaluation framework for maternal-health interactions in Indian contexts that operationalises cultural behaviour through ten dimensions of maternal-health reasoning. Using a 26-item survey administered to 102 pregnant and postpartum women from urban and semi-urban North India, we evaluate ten LLMs under profile-conditioned prompting. Our evaluation separates aggregate cultural alignment and context-dependent sensitivity to demographic and household factors. While several proprietary models approximate human population-level distributions, all evaluated systems exhibit substantially reduced behavioural variation across profiles. We further show that human-grounded conditioning enhances the realism and contextual coherence of synthetic maternal health dialogues. Together, our findings highlight the importance of evaluating healthcare LLMs not only for factual correctness but also for their ability to model socially situated reasoning in culturally embedded healthcare settings.
2023
Hindi Chatbot for Supporting Maternal and Child Health Related Queries in Rural India
Ritwik Mishra | Simranjeet Singh | Jasmeet Kaur | Pushpendra Singh | Rajiv Shah
Proceedings of the 5th Clinical Natural Language Processing Workshop
Ritwik Mishra | Simranjeet Singh | Jasmeet Kaur | Pushpendra Singh | Rajiv Shah
Proceedings of the 5th Clinical Natural Language Processing Workshop
In developing countries like India, doctors and healthcare professionals working in public health spend significant time answering health queries that are fact-based and repetitive. Therefore, we propose an automated way to answer maternal and child health-related queries. A database of Frequently Asked Questions (FAQs) and their corresponding answers generated by experts is curated from rural health workers and young mothers. We develop a Hindi chatbot that identifies k relevant Question and Answer (QnA) pairs from the database in response to a healthcare query (q) written in Devnagri script or Hindi-English (Hinglish) code-mixed script. The curated database covers 80% of all the queries that a user of our study is likely to ask. We experimented with (i) rule-based methods, (ii) sentence embeddings, and (iii) a paraphrasing classifier, to calculate the q-Q similarity. We observed that paraphrasing classifier gives the best result when trained first on an open-domain text and then on the healthcare domain. Our chatbot uses an ensemble of all three approaches. We observed that if a given q can be answered using the database, then our chatbot can provide at least one relevant QnA pair among its top three suggestions for up to 70% of the queries.