When FEVER Meets Finance: A Controlled Benchmark Reveals Overlap-Driven Financial Claim Verification

Suprabhat Rapolu


Abstract
Realistic financial misinformation is typically a single-token edit of a true statement, yet no public financial dataset supports per-edit statements about which factual changes a verification system can detect. We introduce a controlled benchmark of 1,489 claim-evidence pairs derived from SEC 10-K filings via documented FactCC-style perturbations (number changes, negation flips, entity swaps, evidence mismatches), each pair tagged with its generation rule, and use it to evaluate eleven models spanning three technological generations - TF-IDF linear classifiers, tree ensembles, and neural models including a fine-tuned transformer - all trained on FEVER under one protocol. Per-edit attribution is our central result: the strongest model verifies verbatim financial claims at 98.8% accuracy but detects single-token falsifications at 33% accuracy or below, with number changes and negation flips detected at similar rates and entity swaps far the hardest (2.6%, on 38 pairs). Two diagnoses follow. A hypothesis-only probe shows the classical tier’s in-domain competence is claim-surface artifact - a claim-only classifier reaches macro-F1 0.556 against 0.549 for full claim+evidence models - while our benchmark shows no such artifact under the same probe (0.358). And transfer to real financial claims collapses: on Fin-Fact, no learned model beats a tuned lexical-overlap heuristic (macro-F1 0.368), despite in-domain scores up to 0.841. We release the benchmark, generation code, and evaluation harness.
Anthology ID:
2026.finnlp-1.42
Volume:
Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Jimin Huang, Yongjae Lee, Sophia Ananiadou, Hsin-Hsi Chen, Hiroya Takamura, Chung-Chi Chen
Venues:
FinNLP | WS
SIG:
SIGFinTech
Publisher:
Association for Computational Linguistics
Note:
Pages:
518–523
Language:
URL:
https://aclanthology.org/2026.finnlp-1.42/
DOI:
Bibkey:
Cite (ACL):
Suprabhat Rapolu. 2026. When FEVER Meets Finance: A Controlled Benchmark Reveals Overlap-Driven Financial Claim Verification. In Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing, pages 518–523, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
When FEVER Meets Finance: A Controlled Benchmark Reveals Overlap-Driven Financial Claim Verification (Rapolu, FinNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.finnlp-1.42.pdf