Vignesh Coumarane
Author directory2026
Valid but Wrong: Domain Invariants and a Cross-Model Plan-Validity Benchmark for Household-Finance LLMs
Vignesh Coumarane
Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing
Vignesh Coumarane
Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing
Today’s LLMs are good at solving scoped financial tasks. A plan may be type-correct, schema-valid, and executable, yet not financially sound: double-spending the same dollar, or leaving free employer-match money on the table. We call these valid but wrong, and show that today’s evaluations do not consider them. We contribute: (1) a validity hierarchy separating syntactic, schematic, and executional validity from domain validity; (2) a succinct list of domain invariants operationalizing this idea for household finance, divided into conservation and efficiency invariants, grounded in a real six-figure failure in a deployed system; (3) a cross-model measure: we ask eleven current LLMs to plan out a balanced scenario bank (N=585 surface-valid plans) and evaluate each using our invariants. All conform to type, schema, and execution, yet a nonzero fraction break domain invariants which escape detection in surface validity checks. We distinguish a dominance violation over capturable match (unassailable) from an instruction-following violation (does-it-sum) and a contested idle-cash violation. Pooled rates of violating unassailable invariants (the "floor") are 15.2% (95% CI 12–18%), driven by two small models and by instruction-following; the dominance-only floor is 6.2% pooled and 3.6% for the nine frontier models, whose per-model floors overlap. However, the best model (GPT-5) breaks no unassailable invariants, but parks cash idle on 41% of plans, while a small deployable model fails basic conservation 85% of the time. A median 2,500 in guaranteed employer match is tossed aside per dominance violation. Floors hold even on trap-free controls (7.6%). Using a tighter definition counting idle cash, which is a preference-based decision, we find an upper bound of 51.3%. Capability improves on conservation errors but does not close the gap on domain validity.