Tichanat Ruangsiri

Author directory

2026

Multilingual agentic benchmarks remain rare, especially for under-resourced languages such as Thai, where nicknames, unspaced script, and fused honorifics change how an agent must query its tools. We present FahMai-TeleBench, a bilingual (Thai/English) benchmark for grounded tool use. A model answers employee-directory questions over a synthetic company directory by issuing tool calls, or refuses when the answer is absent. All 626 items are synthetic but modeled on a real Thai enterprise-directory deployment, and they are graded deterministically without an LLM judge. Our central design choice is to treat tool availability as an experimental variable: each item runs under four tool configurations (grep, structured search, both, and a Python REPL), so the benchmark measures not just whether a model answers but how it uses tools. The task is near-solved at the frontier (GPT-5.4 97–99%), yet sharply discriminating for the Thai-capable models an organization would host itself (Typhoon-2.5 58–74%, Typhoon-S-8B 25–37%). The Thai penalty is model-specific, and Typhoon-2.5 answers 90% of items correctly once the relevant records are placed in its context, so its failures are failures of tool use. The tool axis also exposes differences at the frontier: GPT-5.5 drops to 73% with a wide-schema structured search tool while scoring 97–98% with grep and the REPL.