Timothy Veigel
Author directory2026
ConFormBench: Evaluating Evidence-Supported Record Maintenance in Conversational Form Filling
Timothy Veigel | Sven Mayer
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Timothy Veigel | Sven Mayer
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Conversational form-filling systems must update structured records across dialogue turns. A user may add, correct, or retract information, while the existing record may itself contain stale or unsupported values. We call correct maintenance of this state record commitment fidelity: after each turn, the complete record should contain all and only commitments supported by the schema and dialogue history. We introduce ConFormBench, a diagnostic benchmark that tests this capability. Each item provides a form schema, a prior record, dialogue history, and a new user utterance; the system must return the complete updated record. ConFormBench contains 180 source-grounded synthetic update turns across four institutional form families, covering 339 scoreable fields and 1,683 changed leaf-field commitments. We mechanically validate all items and audit a coverage-controlled 60-item subset through blind independent reconstruction and discrepancy adjudication. We also audit the evaluator against the adjudicated labels. On the 144-turn held-out split, five systems achieve 95.1-97.0% all-field accuracy, yet only 18.1-33.3% of records are entirely correct. Errors concentrate on clearing values, using history, coordinating gates, routing repeated instances, and repairing prior-state errors. Thus, field-level scores substantially overstate record-maintenance reliability.