Large Language Models are deployed across languages, yet whether their responses to valueladen questions remain stable across prompt languages is unclear. We evaluate sixteen instruction-tuned models, seven in the 7–9B band and nine larger, all open-weight except one. On translated GlobalOpinionQA items we compare answer distributions across seven languages and against country-matched survey pluralities. Model responses vary systematically by prompt language, yet in every language they stay closer to US survey responses than to the country-matched population. On items that carry both a US and a Chinese human answer, all seven smaller models match the US one more often than the Chinese one. In a crossed prompt-language and national-identity cue experiment, responses stay closer to the prompt-language condition than to the statedidentity condition, including in the larger models, even though a cue-uptake probe confirms the identity cue is registered. This demonstrates language dominance over identity cues, although it does not by itself identify training data as the cause. This inconsistency matters most when the request is harmful, since a model judged safe in one language need not be safe in another. On paired English– Chinese prompts from TRIDENT, a benchmark of expert-validated harmful requests, four of the five non-Chinese-origin models in the 7–9B band receive significantly higher harmfulness ratings in Chinese, whereas neither Chineseorigin model does. The language gap replicates on six larger models, though the origin split does not. The choice of translator also changes the measured answers, so we report metrics and comparisons that do not depend on it, and we treat raw cross-lingual distances as uninterpretable on their own.

Recent model editing techniques promise to mitigate the problem of memorizing false or outdated associations during LLM training. However, we show that these techniques can introduce large unwanted side effects which are not detected by existing specificity benchmarks. We extend the existing CounterFact benchmark to include a dynamic component and dub our benchmark CounterFact+. Additionally, we extend the metrics used for measuring specificity by a principled KL divergence-based metric. We use this improved benchmark to evaluate recent model editing techniques and find that they suffer from low specificity. Our findings highlight the need for improved specificity benchmarks that identify and prevent unwanted side effects.

Large language models (LLMs) that do not give consistent answers across contexts are problematic when used for tasks with expectations of consistency–e.g. question-answering, explanations, etc. Our work presents an evaluation benchmark for self-consistency in cases of under-specification where two or more answers can be correct. We conduct a series of behavioral experiments on the OpenAI model suite using an ambiguous integer sequence completion task. We find that average consistency ranges from 67% to 82%, far higher than would be predicted if a model’s consistency was random, and increases as model capability improves. Furthermore, we show that models tend to maintain self-consistency across a series of robustness checks, including prompting speaker changes and sequence length changes. These results suggest that self-consistency arises as an emergent capability without specifically training for it. Despite this, we find that models are uncalibrated when judging their own consistency, with models displaying both over- and under-confidence. We also propose a nonparametric test for determining from token output distribution whether a model assigns non-trivial probability to alternative answers. Using this test, we find that despite increases in self-consistency, models usually place significant weight on alternative, inconsistent answers. This distribution of probability mass provides evidence that even highly self-consistent models internally compute multiple possible responses.

The most interesting words in scientific texts will often be novel or rare. This presents a challenge for scientific word embedding models to determine quality embedding vectors for useful terms that are infrequent or newly emerging. We demonstrate how Latent Semantic Imputation (LSI) can address this problem by imputing embeddings for domain-specific words from up-to-date knowledge graphs while otherwise preserving the original word embedding model. We use the MeSH knowledge graph to impute embedding vectors for biomedical terminology without retraining and evaluate the resulting embedding model on a domain-specific word-pair similarity task. We show that LSI can produce reliable embedding vectors for rare and out-of-vocabulary terms in the biomedical domain.