Junhao Qiu
Author directory2026
BioSciReview-Bench: Benchmarking Coding Agents on Silent Scientific Bug Detection in Bioinformatics Code
Junhao Qiu
Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
Junhao Qiu
Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
Existing coding-agent benchmarks test functional correctness—whether code runs and returns expected outputs. Silent scientific methodology bugs pose a different challenge: they execute without error yet produce plausible, scientifically wrong results. We introduce BioSciReview-Bench, a benchmark of 200 manually reviewed synthetic bioinformatics code snippets (100 matched correct–buggy pairs) spanning five domains and six bug categories. Each snippet is presented in isolation without construction metadata; agents must detect whether a bug exists, localize it to a specific displayed line, and classify it by type. We evaluate six configurations across three model families and two agent harnesses. All achieve detection F1 of 0.94–0.99, but performance drops with stricter requirements: joint localization within three lines reaches 0.92–0.98 F1, and full diagnosis requiring exact line plus correct category falls to 0.74–0.89 F1, a gap of approximately 10–23 percentage points below detection. Filtering and quality-control bugs have the lowest average Full Diagnosis@±3 success rate, and agents sometimes confuse them with normalization errors. Detection performance is high on this controlled benchmark, while delivering actionable diagnoses remains substantially harder.