Debasmita Panda
Author directory2026
Annotating Indian Regional Biases using Large Language Models: Evaluation and Analysis
Debasmita Panda | Akash Anil | Neelesh Kumar Shukla
Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)
Debasmita Panda | Akash Anil | Neelesh Kumar Shukla
Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)
Social biases based on regional identity (or regional bias) are often observed in Indian contexts on major online social networks and require critical attention. However, due to large linguistic and cultural diversity, high annotation costs, and inherent human biases, very little annotated data exists on regional biases in the Indian context. Recently, Large Language Models (LLMs) have garnered attention for the automatic annotation of text. However, such annotation efforts are largely limited to English texts, and LLMs often perform poorly when applied to low-resource languages. Therefore, this paper focuses on understanding the capabilities and challenges of popular open-source LLMs in annotating Indian regional biases. We utilize the recently proposed IndRegBias dataset, which consists of Indian regionally biased social media comments in both English and code-mixed formats. First, we assess the annotation capabilities of LLMs in a zero-shot setting and critically analyze their performance across different writing styles, including code-mixing, transliteration, and English. We find that the majority of LLMs exhibit low agreement with human annotations (measured using Cohen’s kappa). Consequently, we extend our study by fine-tuning the models using 50% of the data and evaluating them on the remaining 50%. We observe a significant improvement in annotation agreement (kappa) for all the LLMs. To further assess the capabilities of the fine-tuned models, we evaluate them on 500 newly collected social media comments discussing regional issues in India. The results show that most fine-tuned LLMs outperform their zero-shot counterparts when annotating these new comments.
IndRegBias: A Dataset for Studying Indian Regional Biases in English and Code-Mixed Social Media Comments
Debasmita Panda | Akash Anil | Neelesh Kumar Shukla
Proceedings of the First Workshop on Pluralistic Value Alignment of LLMs
Debasmita Panda | Akash Anil | Neelesh Kumar Shukla
Proceedings of the First Workshop on Pluralistic Value Alignment of LLMs
While social biases corresponding to gender, race, socio-economic conditions etc., have been extensively studied in the major applications of Natural Language Processing (NLP), biases corresponding to regions have garnered less attention. This is mainly because of (i) difficulty in extraction of regional biased dataset, (ii) disagreements in annotation due to inherent human’s biases, and (iii) regional biases are studied in combination with other types of social biases and are often under-represented. This paper focuses on creating a dataset IndRegBias, consisting of regional biases in an Indian context reflected by user’s comments on popular social media platforms, namely Reddit and YouTube. We carefully selected 25000 comments appearing on various threads in Reddit and videos on YouTube discussing trending topics on regional issues in India. Furthermore, we propose a multilevel annotation strategy to annotate the comments describing the severity of regional biased statements. To detect the presence of regional bias and their severity in IndRegBias, we evaluate open source Large Language Models (LLM) and Indic Language Models (ILM) by using Zero-Shot, Few-Shot, and Fine-Tuning strategies. We observe that Zero-Shot and Few-Shot approaches show lower accuracy in detecting regional biases and severity in majority of the LLMs and ILMs. However, Fine-Tuning approach significantly enhances the performance of the LLM in detecting Indian regional bias along with their severity.
Search