Zhuan Anne Zhang
Author directory2026
Towards a Benchmark for Fire Safety Code Interpretation in the Hong Kong AEC Industry
Le Qiu | Zhuan Anne Zhang | Hok Ming Adeline Chan | Emmanuele Chersoni
Proceedings of the First Workshop on Language Technologies for Crisis Preparedness and Response
Le Qiu | Zhuan Anne Zhang | Hok Ming Adeline Chan | Emmanuele Chersoni
Proceedings of the First Workshop on Language Technologies for Crisis Preparedness and Response
Large Language Models (LLMs) are increasingly explored in the AEC (Architecture, Engineering, and Construction) domain, yet their reliability in regulatory tasks remains uncertain.
In this study, we focused on Hong Kong fire safety codes: together with domain experts, we create a localized dataset of multiple-choice (MCQs) and open-ended questions (OEQs) to evaluate 12 widely used LLMs, including both proprietary and open-source models.
Our findings show that proprietary models may achieve 80–90% accuracy on MCQs, while open-source models generally lag behind. On OEQ, humans and LLM judges rated the answers at 50-70% accuracy, suggesting that LLMs in specialized compliance tasks still have large margins of improvement.
In this study, we focused on Hong Kong fire safety codes: together with domain experts, we create a localized dataset of multiple-choice (MCQs) and open-ended questions (OEQs) to evaluate 12 widely used LLMs, including both proprietary and open-source models.
Our findings show that proprietary models may achieve 80–90% accuracy on MCQs, while open-source models generally lag behind. On OEQ, humans and LLM judges rated the answers at 50-70% accuracy, suggesting that LLMs in specialized compliance tasks still have large margins of improvement.