Most existing benchmarks evaluate social bias in large language models (LLMs) using multiple-choice questions (MCQ) grounded in stereotypical contexts. While MCQ formats enable straightforward quantification, how LLMs manifest social bias in task-specific contexts and under different output constraints remains underexplored. We introduce SCENE, a benchmark for measuring the social bias shown by LLMs in scenario-completion tasks. Given short scenarios involving social groups, models must complete key elements of the situation through two tasks: (i) Character Specification, which asks models to assign social attributes to characters, and (ii) Plot Continuation, which asks models to select the most plausible narrative continuation. Combining templates adapted from prior benchmarks with newly collected templates through a university-wide competition, we construct a dataset of over 75,000 Korean instances. Experiments show that bias measurements are highly sensitive to task context and output constraints, even for identical underlying scenarios, suggesting that widely used MCQ-based evaluations may not fully capture certain manifestations of stereotypes. SCENE is publicly available at https://github.com/DSAIL-SKKU/SCENE-EMNLP-2026.

While prior studies have shown that gendered stereotypes can influence LLMs’ outputs, little attention has been paid to how gendered cues affect depression severity assessment in mental health contexts. In this paper, we systematically investigate the impact of gendered cues on LLM-based depression severity prediction, focusing on depressive disorder. To this end, we introduce a counterfactual evaluation framework, where we inject two types of gendered cues, explicit and implicit, into social media posts and analyze how they shift LLMs’ severity predictions. Our results reveal that gendered cues systematically shift LLMs’ severity predictions, increasing the risk of underestimating severe cases that may involve suicidal ideation or self-harm. To explore a possible mitigation direction, we propose FairDep, a symptom-grounded reasoning scaffold designed to reduce LLM sensitivity to gendered cues in depression severity prediction. By guiding models to focus on symptom-level evidence, FairDep improves prediction consistency under gender cue perturbations while preserving sensitivity to severe cases. Our findings show that gender cue sensitivity is not merely a fairness concern, but a safety concern in LLM-based mental health assessment, where even small prediction shifts can reduce sensitivity to severe cases involving suicidal ideation or self-harm. Our code and resources are publicly available.

Small language models (SLMs) are increasingly proposed for educational use because they promise lower cost, offline deployment, and stronger data privacy. We report an exploratory evaluation of three sub-2B-parameter models on example-based decimal-arithmetic tutoring. Across structured interactions, all three produced fluent, confident output that masked unstable pedagogy and frequent mathematical errors even on elementary decimal addition and place-value tasks. Building on these observations, we describe an emerging interaction-based measurement framework intended to support more defensible readiness decisions about SLMs as math tutors.

Small language models can run offline in a classroom, but they make mathematical errors no teacher can afford to miss. This work-in-progress proposes a measurement-grounded professional development model in which upper-elementary teachers use local LLMs to generate and critically evaluate items; an illustrative pilot demonstrates feasibility, teacher-impact study planned.

Meme captioning, the task of generating a sentence that describes the meaning of a meme, is both challenging and important in advancing Computational Meme Understanding (CMU). However, existing research has not explored its decomposition into subtasks or its connections to other CMU tasks. To address this gap, we introduce MemeInterpret, a meme corpus containing meme captions together with corresponding surface messages and relevant background knowledge. Strategically built upon the Facebook Hateful Memes dataset, MemeInterpret is the last piece in a set of corpora that unifies three major categories of CMU tasks for the first time. Extensive experiments on MemeInterpret and connected datasets suggest strong relationships between meme captioning, its two proposed subtasks, and the other two key categories of CMU tasks: classification and explanation. To stimulate further research on CMU, we make our dataset publicly available at https://github.com/npnkhoi/MemeInterpret.

Named Entity Recognition (NER) plays a pivotal role in medical Natural Language Processing (NLP). Yet, there has not been an open-source medical NER dataset specifically for the Korean language. To address this, we utilized ChatGPT to assist in constructing the KBMC (Korean Bio-Medical Corpus), which we are now presenting to the public. With the KBMC dataset, we noticed an impressive 20% increase in medical NER performance compared to models trained on general Korean NER datasets. This research underscores the significant benefits and importance of using specialized tools and datasets, like ChatGPT, to enhance language processing in specialized fields such as healthcare.

Pre-trained large-scale language models (LLMs) excel at producing coherent articles, yet their outputs may be untruthful, toxic, or fail to align with user expectations. Current approaches focus on using reinforcement learning with human feedback (RLHF) to improve model alignment, which works by transforming coarse human preferences of LLM outputs into a feedback signal that guides the model learning process. However, because this approach operates on sequence-level feedback, it lacks the precision to identify the exact parts of the output affecting user preferences. To address this gap, we propose a method to enhance LLM alignment through fine-grained token-level supervision. Specifically, we ask annotators to minimally edit less preferred responses within the standard reward modeling dataset to make them more favorable, ensuring changes are made only where necessary while retaining most of the original content. The refined dataset is used to train a token-level reward model, which is then used for training our fine-grained Proximal Policy Optimization (PPO) model. Our experiment results demonstrate that this approach can improve LLM performance by up to 5.1% in terms of win rate against the reference model, compared with the traditional PPO model.