Md Mahbub Alam
Author directoryOther people with similar names: Md. Mahbub Alam
2026
REFRAME: A Test Suite for the Knowledge–Deployment Gap in LLMs
Elahe Rahimi | François Roewer-Després | Hassan Sajjad | Md Mahbub Alam | Sher Badshah | Elham Dolatabadi | Frank Rudzicz
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Elahe Rahimi | François Roewer-Després | Hassan Sajjad | Md Mahbub Alam | Sher Badshah | Elham Dolatabadi | Frank Rudzicz
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Large language models (LLMs) are commonly evaluated on multiple-choice tasks that ask them to select an answer or to abstain on unanswerable questions. Neither setting captures frame-breaking: where the question is answerable, but the correct answer lies outside the framing of a candidate set of invalid answers. A useful model must reframe the task, rather than choose an invalid frame-compliant answer. We introduce REFRAME, a test suite of 150 scenarios in which a commonsense constraint invalidates their framing. Each REFRAME scenario is additionally paired with a direct knowledge probe of the same frame-breaking constraint, which we leverage to separate what a model knows from what it deploys. Across nine frontier LLMs, we observe a substantial knowledge-deployment gap with zero-shot prompting. A targeted reasoning prompt only partially closes this gap. Human annotators, by contrast, readily detect the constraint and reframe the problem. Together, these results characterize frame-breaking as a distinct reasoning challenge and demonstrate significant room for LLM improvement.