Contextual question answering (CQA) involves uncertainty regarding the use of available knowledge in forming an answer. In CQA, large language models (LLMs) should decide whether to Refer to explicitly stated information, Infer from contextual evidence using commonsense knowledge, or Abstain when an answer cannot be supported. We investigate whether LLMs implicitly select among these skills under CQA uncertainty. Combining prompting analysis with layer-wise probing, we examine latent skill selection across instruction-tuned LLMs. We find that the models distinguish Refer from Infer more easily than Infer from Abstain. In particular, erroneous abstentions disproportionately occur on inference-based questions, and the Infer–Abstain distinction is the least separable in model representations. We further find that models represent contextual knowledge gaps more strongly than contextual knowledge sufficiency. These results characterize the emergence and limitations of latent skill selection in LLMs under CQA uncertainty.

Large language models have demonstrated varying levels of competence across a range of reasoning tasks, but coarse-grained evaluations often do not reflect their specific strengths and weaknesses, particularly in complex tasks such as Narrative Question Answering. In this paper, we advocate for a multi-dimensional skill-based evaluation that assesses models across distinct core skill dimensions. Our proposed skill-focused evaluation framework offers a granular and more realistic measure of model performance, revealing targeted areas for improvement and guiding future development. Experiments on Narrative Question Answering demonstrate that dimension-level analysis captures the multifaceted nature of the task and informs more effective model evaluation.

Narrative Question Answering is an important task for evaluating and improving reading comprehension abilities in both humans and machines. However, there is a lack of consensus on the skill taxonomy that would enable systematic and comprehensive assessment and learning of the various aspects of Narrative Question Answering. Existing task-level skill views oversimplify the multidimensional nature of tasks, while question-level taxonomies face issues in evaluation and methodology. To address these challenges, we introduce a more inclusive skill taxonomy that synthesizes and redefines narrative understanding skills from previous taxonomies and includes a generation skill dimension from the answering perspective.

Narrative Why-Question Answering is an important task to assess the causal reasoning ability of systems in narrative settings. Further progress in this domain needs clear identification of challenges related to understanding the causal structure of narration. In this paper, we give an overview of the challenges related to both narrative understanding and why-question answering, because Narrative Why-Question Answering combines the characteristics of these domains. We also identify narrative QA datasets containing why-questions and analyze their characteristics through the lens of these challenges.