Guillermo Gil de Avalle
Author directory2026
DiagFlowBench: Evaluating How Language Models Handle Off-Procedure Inputs in Grounded Diagnostic Dialogue
Guillermo Gil de Avalle | Laura Maruster | Shaina Raza | Christos Emmanouilidis
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Guillermo Gil de Avalle | Laura Maruster | Shaina Raza | Christos Emmanouilidis
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Language models increasingly serve as advisory systems in maintenance operations. To prevent hallucination, established approaches ground these models in procedural documentation, constraining them to prescribed sequences. In practice, however, operators may stray from these steps, requiring models to recognise unscripted, off-procedure utterances. Current benchmarks rarely prioritise this capability. We introduce DiagFlowBench, a dataset of 50 industrial diagnostic flowcharts from a consumer manufacturer converted into 1,676 multi-turn conversations that contrast compliant with off-procedure utterances. Evaluating a panel of ten commercial and open-weight models reveals high variability in abstention rates, with models often selecting a real but contextually inadequate step rather than fabricating facts. The inherent plausibility and authority of this mapped but wrong advice exposes a challenging vulnerability for grounding systems.