Large language models (LLMs) are commonly evaluated on multiple-choice tasks that ask them to select an answer or to abstain on unanswerable questions. Neither setting captures frame-breaking: where the question is answerable, but the correct answer lies outside the framing of a candidate set of invalid answers. A useful model must reframe the task, rather than choose an invalid frame-compliant answer. We introduce REFRAME, a test suite of 150 scenarios in which a commonsense constraint invalidates their framing. Each REFRAME scenario is additionally paired with a direct knowledge probe of the same frame-breaking constraint, which we leverage to separate what a model knows from what it deploys. Across nine frontier LLMs, we observe a substantial knowledge-deployment gap with zero-shot prompting. A targeted reasoning prompt only partially closes this gap. Human annotators, by contrast, readily detect the constraint and reframe the problem. Together, these results characterize frame-breaking as a distinct reasoning challenge and demonstrate significant room for LLM improvement.

We present ACCORD, a framework and benchmark suite for disentangling the commonsense grounding and reasoning abilities of large language models (LLMs) through controlled, multi-hop counterfactuals. ACCORD introduces formal elements to commonsense reasoning to explicitly control and quantify reasoning complexity beyond the typical 1 or 2 hops. Uniquely, ACCORD can automatically generate benchmarks of arbitrary reasoning complexity, so it scales with future LLM improvements. Indeed, our experiments on state-of-the-art LLMs show performance degrading to below random chance with only moderate scaling, leaving substantial headroom for improvement. We release a leaderboard of the benchmark suite tested in this work, as well as code for automatically generating more complex benchmarks.

Reducing communication breakdown is critical to success in interactive NLP applications, such as dialogue systems. To this end, we propose a confusion-mitigation framework for the detection and remediation of communication breakdown. In this work, as a first step towards implementing this framework, we focus on detecting phonemic sources of confusion. As a proof-of-concept, we evaluate two neural architectures in predicting the probability that a listener will misunderstand phonemes in an utterance. We show that both neural models outperform a weighted n-gram baseline, showing early promise for the broader framework.