Sven Mayer

Author directory

2026

Conversational form-filling systems must update structured records across dialogue turns. A user may add, correct, or retract information, while the existing record may itself contain stale or unsupported values. We call correct maintenance of this state record commitment fidelity: after each turn, the complete record should contain all and only commitments supported by the schema and dialogue history. We introduce ConFormBench, a diagnostic benchmark that tests this capability. Each item provides a form schema, a prior record, dialogue history, and a new user utterance; the system must return the complete updated record. ConFormBench contains 180 source-grounded synthetic update turns across four institutional form families, covering 339 scoreable fields and 1,683 changed leaf-field commitments. We mechanically validate all items and audit a coverage-controlled 60-item subset through blind independent reconstruction and discrepancy adjudication. We also audit the evaluator against the adjudicated labels. On the 144-turn held-out split, five systems achieve 95.1-97.0% all-field accuracy, yet only 18.1-33.3% of records are entirely correct. Errors concentrate on clearing values, using history, coordinating gates, routing repeated instances, and repairing prior-state errors. Thus, field-level scores substantially overstate record-maintenance reliability.

2018

Today, we see an ever growing number of tools supporting text annotation. Each of these tools is optimized for specific use-cases such as named entity recognition. However, we see large growing knowledge bases such as Wikipedia or the Google Knowledge Graph. In this paper, we introduce NLATool, a web application developed using a human-centered design process. The application combines supporting text annotation and enriching the text with additional information from a number of sources directly within the application. The tool assists users to efficiently recognize named entities, annotate text, and automatically provide users additional information while solving deep text understanding tasks.