Kelsey Rook
Author directory2026
Characterizing Ambiguity in Financial Document Question Answering
Kelsey Rook | Clare Arrington | Tomek Strzalkowski | Razvan Bunescu
Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing
Kelsey Rook | Clare Arrington | Tomek Strzalkowski | Razvan Bunescu
Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing
Large Language Models (LLMs) have demonstrated strong performance in question answering (QA) over general knowledge. However, naturally occurring ambiguity in text poses significant challenges. Users may omit important contextual information from their queries, available context may be insufficient to answer a question without making assumptions, while frontier models tend to give confident answers rather than ask clarifying questions. Research on ambiguity in QA has focused primarily on factoid and general-knowledge settings that rely on LLM-internal knowledge, leaving a gap in research on domain-specific and document QA.
We examine how natural ambiguity arises in financial document QA, focusing on hybrid documents containing both text and tables. To this aim, we annotate 200 questions sampled from the MultiHiertt dataset and find that 77% of them exhibit some degree of ambiguity, with 52% of that ambiguity at least partially due to the data itself. We propose an extended taxonomy that accounts for ambiguity source (document vs. query) and degree. Upon evaluating frontier LLMs on ambiguity detection, we observe sharp drops in accuracy across the ambiguity spectrum and high false positive rates. Additionally, inter-model agreement is low, suggesting that models lack a consistent notion of ambiguity.
We examine how natural ambiguity arises in financial document QA, focusing on hybrid documents containing both text and tables. To this aim, we annotate 200 questions sampled from the MultiHiertt dataset and find that 77% of them exhibit some degree of ambiguity, with 52% of that ambiguity at least partially due to the data itself. We propose an extended taxonomy that accounts for ambiguity source (document vs. query) and degree. Upon evaluating frontier LLMs on ambiguity detection, we observe sharp drops in accuracy across the ambiguity spectrum and high false positive rates. Additionally, inter-model agreement is low, suggesting that models lack a consistent notion of ambiguity.