Aaditya Anand Thokal
Author directory2026
What Deterministic Verification Actually Fixes in Financial QA: A Diagnostic Analysis of Rule Heterogeneity Across Datasets
Aaditya Anand Thokal
Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing
Aaditya Anand Thokal
Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing
Deterministic, parameter-free post-hoc correction rules are an attractive way to address numerical errors in financial question answering: they are simple, cheap, and fully auditable. We ask whether such rules actually behave like general-purpose correctors, or whether their effect is contingent on properties of the dataset and model output distribution they are applied to. We evaluate a two-rule deterministic verification layer (DVL) — scale_mul100 and scale_div100 — on FinQA (n=873) and TAT-QA (n=718) using Mistral-7B-Instruct-v0.3. This same, fixed model-plus-rule configuration produces a striking cross-dataset divergence: harmful on FinQA (37.34%→16.04%, −21.31pp, McNemar p=1.76×10⁻³⁹) and beneficial on TAT-QA (26.32%→30.64%, +4.32pp, p=5.15×10⁻³). A rule-level transition analysis shows scale_mul100 is highly precise on TAT-QA (87.95%) but overwhelmingly imprecise on FinQA (3.39%), while scale_div100 produces zero observed corrections and only harm in both settings; removing it raises TAT-QA accuracy further, to 35.10%. We do not find evidence that these rules correct arithmetic, retrieval, or semantic reasoning failures. We conclude that this specific pair of scaling rules is better understood as a targeted, dataset-contingent representation normalizer than as a general reasoning corrector, and we characterize the boundary conditions under which it helps or hurts.