Kazutaka Yamasaki
Author directory2026
TRUST-BENCH: A Synthetic Japanese Financial-Group Benchmark for Enterprise Document Access in Coding Agents
Shamik Kundu | Anand Mekala | Kazutaka Yamasaki
Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing
Shamik Kundu | Anand Mekala | Kazutaka Yamasaki
Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing
In large financial institutions, what code should do is rarely recoverable from code alone: the authoritative rules live in internal business documents that are largely unreachable to coding agents. Benchmarks built on public repositories therefore cannot measure what reaching those documents is worth. We present TRUST-BENCH, a fully synthetic benchmark that measures the effect of enterprise document availability and of the channel through which it is delivered: the codebase of a fictional Japanese financial group-1,105 programs and 287k lines of code across five languages, a legacy COBOL/PL/I/RPG core of record surrounded by the Java and C periphery paired with 149 Japanese business documents and 93 maintenance tasks. Tasks are graded two ways: an LLM judge scores answers for correctness and completeness, and generated code is compiled, run, and checked against curated test cases.
We describe the dataset-generation procedure in detail and report results over four context conditions: code only (the baseline); code with the raw documents on disk; code with documents behind a hybrid retrieval tool; and code with documents behind a knowledge-graph tool. Graph-mediated document access raised retrieval-task correctness from 0.68 to 0.94 while the strict build-task pass rate rose from 27% to 91%.
We describe the dataset-generation procedure in detail and report results over four context conditions: code only (the baseline); code with the raw documents on disk; code with documents behind a hybrid retrieval tool; and code with documents behind a knowledge-graph tool. Graph-mediated document access raised retrieval-task correctness from 0.68 to 0.94 while the strict build-task pass rate rose from 27% to 91%.