We separate answering from abstention in a context-based question answering task, where a language model answers a question about a document in a possibly different language or abstains when the document contains no answer. A LoRA adapter on Qwen3.5-9B, trained only on answerable questions, generates the answer, while a linear probe on the frozen base model’s activations decides answerability independently and returns the required refusal string when it predicts none exists. This isolates the model’s answerability signal from its instruction-following. The probe’s ranking of answerable against unanswerable questions transfers across languages, corpora, and negative constructions it was not trained on, but its decision threshold does not; training on in-domain multilingual data, rather than recalibrating the threshold, closes most of that gap. For the adapter, how the answer target is constructed trades off automatic correctness metrics against well-formedness and answer length.

Automatic detection of toxic language has the potential to considerably improve engagement with online spaces. Previous work has characterized toxic language detection as a classification problem, often using fine-grained classes for increased explainability. In this position paper, we argue for a particular way of operationalizing categories of toxic language. Our approach focuses on what is expressed or implied, and breaks down implications based on two traits: (i) the core content of what was expressed, and (ii) relevant stakeholders’ attitudes towards that content. We argue for an approach, which we call toxic reasoning, where such distinctions are made explicit. We point out the benefits for such an approach, and develop a toxic reasoning schema, which can explain categories of toxic language from diverse sources. We demonstrate this by mapping the classes of existing toxic language datasets to the schema. Toxic reasoning promises to provide improved understanding of implicit toxicity while increasing explainability.

Contrastive interpretability methods use pairs (or tuples) of inputs whose elements differ in minimal but controlled ways. Contrast-Consistent Search (CCS) is one such method that can identify whether (language) models encode binary features, such as sentence truth, in their internal activations. One advantage is that this requires no labels, meaning there is no assumption that internal activations agree with those labels, as is the case for supervised methods. In this work, we (1) reproduce results previously obtained with CCS, (2) better understand CCS’s objective, (3) propose Tuple-Contrastive PCA (tcPCA), a method with closed-form solutions, as a spiritual successor, and (4) show how tcPCA has wider applicability than CCS by: allowing for n-ary rather than just binary variables; and, finding contrastive subspaces instead of directions. See https://contrast.stefanfs.me for code and interactive figures.

Natural language reasoning plays an increasingly important role in improving language models’ ability to solve complex language understanding tasks. An interesting use case for reasoning is the resolution of context-dependent ambiguity. But no resources exist to evaluate how well Large Language Models can use explicit reasoning to resolve ambiguity in language. We propose to use ambiguous definite descriptions for this purpose and create and publish the first benchmark dataset consisting of such phrases. Our method includes all information required to resolve the ambiguity in the prompt, which means a model does not require anything but reasoning to do well. We find this to be a challenging task for recent LLMs. Code and data available at: https://github.com/sfschouten/exploiting-ambiguity

In this work we analyze the named entity representations learned by Transformer-based language models. We investigate the role entities play in two tasks: a language modeling task, and a sequence classification task. For this purpose we collect a novel news topic classification dataset with 12 topics called RefNews-12. We perform two complementary methods of analysis. First, we use diagnostic models allowing us to quantify to what degree entity information is present in the hidden representations. Second, we perform entity mention substitution to measure how substitute-entities with different properties impact model performance. By controlling for model uncertainty we are able to show that entities are identified, and depending on the task, play a measurable role in the model’s predictions. Additionally, we show that the entities’ types alone are not enough to account for this. Finally, we find that the the frequency with which entities occur are important for the masked language modeling task, and that the entities’ distributions over topics are important for topic classification.