Hoang-An Vo

Author directory

2026

Existing SQL-based benchmarks usually assume that retrieved records are equally informative. This assumption is limiting for investigative domains, where truthful records may differ substantially in inferential strength. To address this gap, we introduce Holmes-SQL, the first benchmark for urban crime investigation over relational databases. Each case begins with partial cues and requires agents to reconstruct the crime scene, victim, suspect pool, and culprit through database interaction. Holmes-SQL combines state uncertainty, unequal evidential strength, and fallacy-aligned suspect construction to evaluate evidence-aware investigative reasoning. Experiments show that agents achieve over 97% SQL execution success, yet the best solve rate reaches only 49%, highlighting a distinct challenge for evidence-aware investigative reasoning. A further comparison shows that agents consistently underperform when investigating compared with reasoning over all evidence provided upfront, highlighting the added challenge of investigation.