Vignesh Coumarane

Author directory

2026

Today’s LLMs are good at solving scoped financial tasks. A plan may be type-correct, schema-valid, and executable, yet not financially sound: double-spending the same dollar, or leaving free employer-match money on the table. We call these valid but wrong, and show that today’s evaluations do not consider them. We contribute: (1) a validity hierarchy separating syntactic, schematic, and executional validity from domain validity; (2) a succinct list of domain invariants operationalizing this idea for household finance, divided into conservation and efficiency invariants, grounded in a real six-figure failure in a deployed system; (3) a cross-model measure: we ask eleven current LLMs to plan out a balanced scenario bank (N=585 surface-valid plans) and evaluate each using our invariants. All conform to type, schema, and execution, yet a nonzero fraction break domain invariants which escape detection in surface validity checks. We distinguish a dominance violation over capturable match (unassailable) from an instruction-following violation (does-it-sum) and a contested idle-cash violation. Pooled rates of violating unassailable invariants (the "floor") are 15.2% (95% CI 12–18%), driven by two small models and by instruction-following; the dominance-only floor is 6.2% pooled and 3.6% for the nine frontier models, whose per-model floors overlap. However, the best model (GPT-5) breaks no unassailable invariants, but parks cash idle on 41% of plans, while a small deployable model fails basic conservation 85% of the time. A median 2,500 in guaranteed employer match is tossed aside per dominance violation. Floors hold even on trap-free controls (7.6%). Using a tighter definition counting idle cash, which is a preference-based decision, we find an upper bound of 51.3%. Capability improves on conservation errors but does not close the gap on domain validity.
Search
Co-authors
    Venues
    Fix author