Suprabhat Rapolu
Author directory2026
When FEVER Meets Finance: A Controlled Benchmark Reveals Overlap-Driven Financial Claim Verification
Suprabhat Rapolu
Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing
Suprabhat Rapolu
Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing
Realistic financial misinformation is typically a single-token edit of a true statement, yet no public financial dataset supports per-edit statements about which factual changes a verification system can detect. We introduce a controlled benchmark of 1,489 claim-evidence pairs derived from SEC 10-K filings via documented FactCC-style perturbations (number changes, negation flips, entity swaps, evidence mismatches), each pair tagged with its generation rule, and use it to evaluate eleven models spanning three technological generations - TF-IDF linear classifiers, tree ensembles, and neural models including a fine-tuned transformer - all trained on FEVER under one protocol. Per-edit attribution is our central result: the strongest model verifies verbatim financial claims at 98.8% accuracy but detects single-token falsifications at 33% accuracy or below, with number changes and negation flips detected at similar rates and entity swaps far the hardest (2.6%, on 38 pairs). Two diagnoses follow. A hypothesis-only probe shows the classical tier’s in-domain competence is claim-surface artifact - a claim-only classifier reaches macro-F1 0.556 against 0.549 for full claim+evidence models - while our benchmark shows no such artifact under the same probe (0.358). And transfer to real financial claims collapses: on Fin-Fact, no learned model beats a tuned lexical-overlap heuristic (macro-F1 0.368), despite in-domain scores up to 0.841. We release the benchmark, generation code, and evaluation harness.