Reetam Biswas

Author directory

2026

We present a case study developing an ontology-guided framework for extracting and validating procedural knowledge from safety-critical documents with limited annotations, specifically the Lockout-Tagout (LOTO) procedural documents. We build multimodal Procedural Knowledge Graphs (PKGs) grounded in the Procedural Knowledge Ontology (PKO) from text and diagrams using LLMs and VLMs. The extracted PKGs are evaluated on instruction- and graph-level metrics with validation against a domain expert curated ontology for action–entity relations, step order, and cross-modal alignment. Embedding-based metrics capture semantic alignment, while lexical metrics expose safety-critical mismatches. Our experiments on proprietary and public LOTO documents demonstrate the feasibility and limitations of multimodal procedural extraction and motivate ontology-guided validation.