While in-context learning is generally shown to be effective for Large Language Models (LLMs), misleading or flawed contexts can cause performance degradation and mode collapse, an empirical phenomenon we term "pigeonholing". Unintentionally bad contexts can happen without jailbreak intent. For example, a user prompts the model to justify an incorrect math theorem or fails to correct buggy agent code. Specifically, we investigate pigeonholing in two settings: (1) when a user suggests an incorrect solution, and (2) when the multi-turn dialogue history contains the assistant’s incorrect answers from prior turns. Our evaluations across 10 verifiable and open-ended tasks with 10 different models show that pigeonholing manifests in three distinct ways: (1) repeating the incorrect answers from the context (causing a 38-40% performance drop), (2) reducing answer diversity in open-ended text generation, and (3) reversing stances on controversial topics to align with the previous user or assistant’s perspective. Further, we find that pigeonholing worsens almost monotonically with context length (i.e., conversation turns). Finally, while standard RLVR-trained models show improvements on clean prompts, they fail to maintain performance on corrupted contexts. We take a step towards mitigating model vulnerability by training with RLVR on synthetic errors, improving model performance on bad contexts by 43–60% over vanilla baselines.

Large language models (LLMs) are increasingly used to produce open-ended, interpretive annotations, yet there is no validated, scalable measure of idea-level similarity to expert annotations. We (i) introduce the content evaluation of LLM annotations as a core, understudied task, (ii) propose IDEAlign for capturing expert similarity judgments via the odd-one-out tasks, and (iii) benchmark various similarity methods, such as text embeddings, topic models, and LLM-as-a-judge, against these human ratings. Applying this approach to two real-world educational datasets (interpreting math reasoning and feedback generation), we find that most metrics fail to capture the nuanced dimensions of similarity meaningful to experts. LLM-as-a-judge performs best (11–18% improvement over other methods) but still falls short of expert alignment, making it useful as a triage filter rather than a substitute for human review. Our work demonstrates the difficulty of evaluating open-ended LLM annotations at scale, and positions IDEAlign as a reusable protocol for benchmarking on this task, thereby informing responsible deployment of LLMs.

We present EduCoder, an open-source web platform designed for annotating classroom conversation transcripts. Existing annotation tools do not support the team-based workflows or access to instructional context that education discourse research requires. EduCoder addresses these gaps by combining transcript text, synchronized video, and instructional materials within a single workspace. The platform supports scoping annotation to specific portions of a lesson, coordinating work across annotation teams, and optionally integrating LLM-generated annotations with structured human–LLM comparison. EduCoder is freely accessible at https://edu-coder.com.